jamesob
Featured
local-llm
A comprehensive guide for building and configuring a high-end local machine to run state-of-the-art LLMs, with detailed hardware choices, BIOS tuning, and Docker-based model serving.
GLM-5.2-QuantTrio-200K-4x-DGX-Spark
A recipe to serve the full-quality, unpruned GLM-5.2-Int4-Int8Mix checkpoint (256 experts) on a 4× NVIDIA DGX Spark cluster, achieving 36 tok/s peak single-stream and 200K context with MTP speculative decoding and CUDA graphs.
Stars
72
7-day growth
No data
Forks
8
Open issues
2
License
Apache-2.0
Last updated
2026-07-25
AI repository intelligence
FR-AI / ANALYSISWhy it is worth attention
Delivers near-lossless inference of a large Mixture-of-Experts model on modest hardware (4× GB10) with detailed, battle-tested setup instructions and open-source contributions back to the community.
Who it is for
- Developers running GLM-5.2 on multi-node DGX Spark clusters
- Researchers seeking high-throughput, long-context serving of unpruned MoE models
- System integrators and DevOps engineers deploying vLLM with custom kernels on ARM-based GB10 nodes
- Enthusiasts reproducing state-of-the-art inference performance on consumer-grade hardware
Use cases
- Serving full-quality GLM-5.2 for long-document analysis (up to 200K tokens)
- High-throughput multi-concurrent inference (6 streams, 65–75 tok/s aggregate)
- Interactive applications requiring low first-token latency (0.36 s with thinking disabled)
- Reproducing and extending benchmark results on 4× GB10 clusters
Strengths
- Achieves 32.5 tok/s mean single-stream and 65–75 tok/s aggregate on 4× GB10
- Supports 200K context with fp8 sparse-MLA KV cache and full CUDA graphs
- Uses in-checkpoint MTP drafter – no separate download or version matching
- Provides reproducible, step-by-step setup with vendored kernels and patches to avoid common issues
Considerations
- Requires exactly a 4-node DGX Spark cluster with RoCE fabric and jumbo frames
- Setup involves manual, multi-step process (custom Docker build, kernel patching, NCCL staging) with many pitfalls
- Long-context requests can trigger a Triton fault that kills the engine; self-heal mitigation requires ~15–20 min reload
README quick start
Quick start
Description
Recipe: GLM-5.2 (unpruned QuantTrio Int4-Int8Mix) at 200K ctx with MTP spec decode on a 4x NVIDIA DGX Spark (GB10) cluster
Related repositories
Similar projects matched by category, topics, and programming language.
bjan
Featured
claudemux
Claudemux is a lightweight, dependency-minimal message bus that lets multiple Claude Code sessions communicate in real-time over tmux, enabling one session to ask another a question and receive the answer without manual intervention.
AARomanov1985
Audio-Cassette-Simulation
A set of bash scripts using ffmpeg to simulate vintage cassette tape audio profiles with noise, wow/flutter, bandwidth limits, and EQ.