A patch that enables correct in-server continuous-batch concurrency for DSpark speculative decoding on NVIDIA DGX Spark, overcoming the default max-num-seqs=1 limitation.
11
No data
0
0
Apache-2.0
2026-06-29
It removes a hard concurrency bottleneck in DSpark serving, delivering up to 290 tok/s with 16 concurrent requests and zero errors, while preserving single-stream byte-identical output.
Correct in-server continuous-batch concurrency for DSpark speculative decoding on NVIDIA DGX Spark (GB10 / SM121).
Stock DSpark serving forces --max-num-seqs 1 — the upstream recipe states the DSpark draft path "stalls under max_num_seqs>1 in this build, so requests serialize. The win is depth (context), not width (concurrency)."
This patch removes that limitation. With it, DSpark serves multiple concurrent streams correctly under independent (staggered) request arrivals, while keeping single-stream output byte-identical to the unpatched engine.
Validated on DeepSeek-V4-Flash-DSpark, vLLM DSpark overlay (Rafael Caricio integration / TonyD2Wild packaging), TP=2 on 2× DGX Spark (GB10), 2026-06-29. The patch is in the model-agnostic DSpark proposer/model code, so it is expected to apply to DeepSeek-V4-Pro-DSpark as well — but Pro is untested; please report results.
Headline below; full single + dual-stack tables in RESULTS.md.
One stack = 2× DGX Spark (GB10), TP=2, kv-cache-dtype fp8, DSpark γ=5,
gpu-memory-utilization 0.80, max-num-seqs 16.
| total concurrency | static (best-case) | staggered (real arrivals) | acceptance | errors |
|---|---|---|---|---|
| 1 | 49 tok/s | ~50 tok/s | ~0.6 | 0 |
| 4 | 122 tok/s | 104 tok/s | ~0.59 | 0 |
| 8 | 183 tok/s | 139 tok/s | ~0.56 | 0 |
| 16 | 290 tok/s | 191 tok/s | ~0.55 | 0 |
| total concurrency | 1 stack | 2 stacks | scaling |
|---|---|---|---|
| 32 | — | 375 tok/s (16+16, 32/32 ok) | ~1.96× |
Replicas scale ~linearly (independent stacks behind a least-connections router).
A deterministic request (temperature 0, ignore_eos) produces byte-identical output whether run alone or while other requests start/finish around it (forcing continuous-batch condense). Run benchmarks/correctness_test.py.
Two root causes blocked max_num_seqs>1:
Patch 1 — request-stable KV slot. DSpark's only persistent pe
Concurrency Patch to enable on DSpark - DeepSeek V4 Flash/Pro C=16 can be modified for more
Similar projects matched by category, topics, and programming language.
Harness Engineering is a methodology for improving coding agent outputs by carefully crafting the environment around them—providing curated context, tools, and executable constraints that encode an organization’s nonfunctional requirements and cumulative lessons.
A 28.9 million parameter language model runs on an $8 ESP32-S3 microcontroller entirely on-device, generating simple stories at about 9.5 tokens per second.
mimic captures traffic from any iOS or web app and automatically generates a Python client library that lets you call the app's API like a regular library.