它消除了 DSpark 服务中的硬并发瓶颈，在 16 个并发请求下达到 290 tok/s 且零错误，同时保持单流输出字节一致。
Correct in-server continuous-batch concurrency for DSpark speculative decoding on NVIDIA DGX Spark (GB10 / SM121).
Stock DSpark serving forces --max-num-seqs 1 — the upstream recipe states the DSpark draft path "stalls under max_num_seqs>1 in this build, so requests serialize. The win is depth (context), not width (concurrency)."
This patch removes that limitation. With it, DSpark serves multiple concurrent streams correctly under independent (staggered) request arrivals, while keeping single-stream output byte-identical to the unpatched engine.
Validated on DeepSeek-V4-Flash-DSpark, vLLM DSpark overlay (Rafael Caricio integration / TonyD2Wild packaging), TP=2 on 2× DGX Spark (GB10), 2026-06-29. The patch is in the model-agnostic DSpark proposer/model code, so it is expected to apply to DeepSeek-V4-Pro-DSpark as well — but Pro is untested; please report results.
Headline below; full single + dual-stack tables in RESULTS.md.
One stack = 2× DGX Spark (GB10), TP=2, kv-cache-dtype fp8, DSpark γ=5,
gpu-memory-utilization 0.80, max-num-seqs 16.
| total concurrency | static (best-case) | staggered (real arrivals) | acceptance | errors |
|---|---|---|---|---|
| 1 | 49 tok/s | ~50 tok/s | ~0.6 | 0 |
| 4 | 122 tok/s | 104 tok/s | ~0.59 | 0 |
| 8 | 183 tok/s | 139 tok/s | ~0.56 | 0 |
| 16 | 290 tok/s | 191 tok/s | ~0.55 | 0 |
| total concurrency | 1 stack | 2 stacks | scaling |
|---|---|---|---|
| 32 | — | 375 tok/s (16+16, 32/32 ok) | ~1.96× |
Replicas scale ~linearly (independent stacks behind a least-connections router).
A deterministic request (temperature 0, ignore_eos) produces byte-identical output whether run alone or while other requests start/finish around it (forcing continuous-batch condense). Run benchmarks/correctness_test.py.
Two root causes blocked max_num_seqs>1:
Patch 1 — request-stable KV slot. DSpark's only persistent pe
Concurrency Patch to enable on DSpark - DeepSeek V4 Flash/Pro C=16 can be modified for more
根据分类、Topic 和编程语言匹配的相似项目。
Harness Engineering is a methodology for improving coding agent outputs by carefully crafting the environment around them—providing curated context, tools, and executable constraints that encode an organization’s nonfunctional requirements and cumulative lessons.
A 28.9 million parameter language model runs on an $8 ESP32-S3 microcontroller entirely on-device, generating simple stories at about 9.5 tokens per second.
mimic captures traffic from any iOS or web app and automatically generates a Python client library that lets you call the app's API like a regular library.
