Doesn't fit Spark
Such a shame that a box marketed as "run large LLMs locally" can't fit this even in NVFP4 in a reasonable setup.
VLLM_PLE_MMAP=1 ; VLLM_PLE_MMAP_WORKERS =32; VLLM_PLE_MMAP_WORKERS=auto;
PLE mmap: layer 1, 128 shards, 320001536 rows x 160 B (47.7 GiB on disk), dtype F8_E4M3, 32 workers
Thanks for the response. Claude tells me this requires some umerged PRs in vLLM to work (54129). Is he halucinating?
It can fit but Nvidia has left the DGX Spark community to fend for themselves it seems
@donghwan-shin so essentially we should opt for RadixArk/Qwen3.8-Flash-Next-NVFP4 is what you are saying or is the SGL also important?
I was going to use vLLM.
@turowicz I went for the RadixArk/Qwen3.8-Flash-Next-NVFP4 as a base and used the nvidia model optimizer to shape it for the DGX Spark so that I can run it well on vLLM.
Seems like the custom recipes are the only option for now ¯_(ツ)_/¯
by "shape it" you mean "tune run args"? care to share them?
@donghwan-shin so essentially we should opt for
RadixArk/Qwen3.8-Flash-Next-NVFP4is what you are saying or is the SGL also important?
I was going to use vLLM.
I mainly followed the stable recipe from SGLang (with docker pull lmsysorg/sglang:dev-qwen38-next-local):
# PLE table -> sparse 47.7 GiB file under $SGLANG_CACHE_DIR/ple/<model> (put it on local NVMe; --ple-offload-dir relocates it).
# Delete the previous table file before each boot until the rewrite is fixed upstream: rewriting a populated file runs at ~17 MB/s (~55 min), a fresh sparse file at GB/s (~8 min).
sglang serve \
--model-path nvidia/Qwen3.8-Flash-Next-NVFP4 \
--tp 1 \
--moe-runner-backend flashinfer_cutlass \
--fp4-gemm-backend flashinfer_cutlass \
--page-size 64 \
--chunked-prefill-size 4096 \
--context-length 262144 \
--speculative-algorithm NEXTN \
--speculative-num-steps 3 \
--speculative-eagle-topk 1 \
--speculative-num-draft-tokens 4 \
--max-running-requests 4 \
--max-mamba-cache-size 20 \
--reasoning-parser qwen3 \
--tool-call-parser auto \
--mem-fraction-static 0.85 \
--host 0.0.0.0 \
--port 30000 \
--ple-offload-embedding \
--ple-offload-backend file
The only change I made was to reduce --max-running-requests to 4 according to my work pattern (using a low number of long-horizon subagents).
Apparently, SGLang is as good as vLLM (at least in my workflow), and ple-offload is essential to make it runnable on a single DGX.
It ABSOLUTELY fits a Spark with the n-gram table offloading to NVME, I have one running that on my desk right now. With tuning and a big bag of tricks applied, it even gets 60 tok/s on structured/code decoding and 40 tok/sec on prose which is really, really not bad at all and usable. And the results are really good, I'm impressed, even side by side to bigger glm models.
I want to share my docker-compose.yml
In order to make PLE working you have to use nvidia-driver-580-open - the KEY word here is open
I haven't tested it with other drivers but it works awesome with 580.178.04
I'm getting 4 parallel streams with ~75tok/s for token generation (aggregated 4 streams) and ~2000tok/s for prompt processing
services:
app:
image: lmsysorg/sglang:latest
# Delete stale PLE tables on every container start (compose up, crash
# restarts, host reboot) so the PLE fill always takes the fresh-sparse
# fast path (~8 min) instead of rewriting a populated 48 GiB file (~55 min),
# then chain to the image's original entrypoint. "$@" = the command below.
entrypoint:
- bash
- -c
- rm -f /ple-cache/ple_table_*.bin; exec /opt/nvidia/nvidia_entrypoint.sh "$@"
- "--"
volumes:
- "./models:/root/.cache"
restart: always
privileged: true
ports:
- 8080:8080
command: sglang serve
--model-path nvidia/Qwen3.8-Flash-Next-NVFP4
--tp 1
--moe-runner-backend flashinfer_cutlass
--fp4-gemm-backend flashinfer_cutlass
--page-size 64
--chunked-prefill-size 4096
--context-length 262144
--kv-cache-dtype fp8_e4m3
--mamba-radix-cache-strategy extra_buffer_lazy
--max-running-requests 6
--max-mamba-cache-size 20
--reasoning-parser qwen3
--tool-call-parser qwen3_coder
--mem-fraction-static 0.85
--reasoning-parser auto
--tool-call-parser auto
--speculative-algorithm NEXTN
--speculative-num-steps 3
--speculative-eagle-topk 1
--speculative-num-draft-tokens 4
--host 0.0.0.0
--port 8080
--ple-offload-embedding
--ple-offload-backend file
--ple-offload-dir /ple-cache
--enable-metrics
ulimits:
memlock: -1
stack: 67108864
ipc: host
deploy:
resources:
reservations:
devices:
- driver: nvidia
count: all
capabilities: [gpu]
UPD 2026-10-09:
- Updated
image: lmsysorg/sglang:dev-qwen38-next-localtoimage: lmsysorg/sglang:latest - Updated
--max-running-requests 3to--max-running-requests 6 - Added
--kv-cache-dtype fp8_e4m3
This allows to handle up to ~1.7M tokens for 6 sessions in parallel
I'm getting around 80tok/s with 3-6 streams running in parallel
UPD 2026-10-05:
Upgrading NVIDIA drivers to nvidia-driver-610-open 610.57.04 works as expected. No problems observed.
In my setup I use headless GB10 and I was able to push --mem-fraction-static to 0.90. --max-running-requests is 3, 16GB of swap file.
For anyone who wanted to run this model on one Spark with vLLM we published a step-by-step recipe that works.
Build, launch and check scripts are in the repo:
https://github.com/salient-data/qwen38-next-dgx-recipe
⚡ 67.5 tokens/sec for one user
⚡ 90.7 tokens/sec in total, six users at once
✅ 99/100 on a grade-school math accuracy check
These are numbers from our one Spark, measured with reasoning off (the math check with it on); plain prose runs at about 38 to
kens/sec, and the six-user figure is from one run.
Community project, not affiliated with NVIDIA, Qwen or vLLM. Reproduction reports welcome!