LLM serving on NVIDIA DGX Spark

This repository contains notes, launchers, benchmarks, and serving guides for running local LLMs on NVIDIA DGX Spark. It is intended for MVPs and demonstrations and does not discuss production security, availability, or monitoring.

Current model recommendation

The current best general-purpose model tested on a single DGX Spark is Qwen3.8 27B with SGLang, NVFP4, and DFlash2. Two serving paths are validated:

• Containerized DFlash2 depth-16 profile (preferred): approximately 70 tokens/s greedy single-stream decode (69.7 measured median; code probes have reached 87.9) using a calibrated NVFP4 drafter. See section 10 of the SGLang guide.
• Native depth-8 profile: approximately 38 tokens/s in sequential single-request generation and approximately 22 tokens/s per request at 10 concurrent requests.

Sequential uncached prefill has been observed at approximately 1,500 prompt tokens/s; prefill depends strongly on prompt length, cache state, and request concurrency. Both profiles serve an OpenAI-compatible API on port 8082 and only one may run at a time.

The Bonsai model should be dismissed for this use case. Its quality and/or serving performance are not sufficient compared with the Qwen3.8 27B setup, so it should not be considered the preferred DGX Spark model.

With a DGX configuration having more VRAM, DeepSeek V4 Flash 0731 may be a better choice because it has only A10B active parameters, which can provide a stronger quality/throughput trade-off when the larger model footprint fits comfortably in memory. It should be evaluated separately on the higher-memory system rather than used as the default single-Spark recommendation.

Documentation

• llama.cpp serving on DGX Spark
• Qwen3.8 27B SGLang, NVFP4, and DFlash2
• Bonsai 27B installation — retained for reference only; Bonsai is not recommended.

Security warning

Do not expose local model servers to the public internet without additional protection. Several example launchers bind to 0.0.0.0 and do not provide authentication. Use a firewall, VPN, reverse proxy, or another suitable security layer when access is not limited to a trusted local network.

Factory reset and data removal

If the DGX Spark has handled confidential files, remove those files before returning or repurposing the system. There may be better approaches depending on the threat model; consult the appropriate organizational security process.

For a basic file-removal operation, the original notes used:

sudo apt update
sudo apt install wipe
wipe -rfi /home/your_user/confidential_folder

The shred utility may also be available for individual files. For a complete system reset, follow NVIDIA's DGX Spark system recovery procedure.

Interesting links

• Optimize PyTorch for DGX Spark
• NVIDIA guide for connecting two Spark units