slvDev
Featured
esp32-ai
A 28.9 million parameter language model runs on an $8 ESP32-S3 microcontroller entirely on-device, generating simple stories at about 9.5 tokens per second.
AI repository intelligence
FR-AI / ANALYSISWhy it is worth attention
It provides an end-to-end, hash-pinned reproducibility kit for a large vLLM serving stack, including exact GPU ordering, fan policies, benchmark harness, and captured live evidence, so configuration drift is visible and recoverable.
Who it is for
- Operators deploying large GLM models on vLLM with multiple GPUs
- ML infrastructure engineers focused on reproducible serving benchmarks
- Teams using NVIDIA Blackwell GPUs for LLM serving
- Pop!_OS/Ubuntu server administrators validating CUDA graph and MTP workloads
Use cases
- Recreate the validated four-GPU GLM-5.2-EXL3 serving stack on similar hardware
- Run the continuous C1/C2/C4 hot benchmark to measure decode tokens/sec, prompt processing, and TTFT
- Verify that a running host matches the known-good image, process arguments, GPU/CPU policies, and fan settings
- Audit configuration drift with the captured snapshot and Local Studio recipe
Strengths
- Pinned immutable image digest and exact compose files enable reproducible deployment
- Automated scripts cover installation, startup, verification, benchmarking, and live-state capture
- Snapshot evidence includes container inspection, process args, image metadata, hardware inventory, and service state
- Benchmark includes SSE idle and engine-stall watchdogs that preserve logs and restart the pinned container
Considerations
- Requires exactly four visible NVIDIA GPUs and specific model/cache directories, though paths can be overridden
- CPU fan control service is board-specific and validated only on the ASUS Pro WS WRX90E-SAGE SE
- The bundle is pinned to a specific vLLM image and 30 July 2026 validated configuration, so porting requires re-validation
README quick start
GLM-5.2 EXL3 four-GPU reproduction bundle
This bundle reproduces the live Pop!_OS configuration validated on 30 July 2026. It runs GLM-5.2-EXL3 on four RTX PRO 6000 Blackwell GPUs, keeps normal high-reasoning behavior, uses MTP3, and preserves the server's full and piecewise CUDA graphs. It does not use eager execution, disable CUDA graphs, add stop sequences, or impose a generation token cap.
Known-good operating point
- Server image:
voipmonitor/vllm:gilded-gnosis-v20-vllmb46c3aa-si35aebc6-fi801d57a-cu132-20260730-r12@sha256:bcd65c13076b00cf7804001ce70085453de958b082897b9674a26d9fab4d436a - Model:
GLM-5.2-EXL3-TR3-3.0bpw - GPUs: four RTX PRO 6000 Blackwell, 250 W limit each
- CPU: boost enabled, 3.1 GHz maximum,
powersavegovernor,balance_performanceEPP - CPU fan policy: 60% at 20 C, 80% at 40 C, 100% at 55 C and above
- GPU fan policy: 50% at 30 C, linear aggressive ramp, 100% at 80 C
- Safety stops in benchmark: CPU 88 C, GPU 82 C
- Serving: TP4, DCP4,
ag_rs, MTP3 greedy - KV cache: NVFP4 DSA MLA KV, configured by the pinned image launcher
- Context: 140,000 maximum model length
- Scheduler: maximum 8 sequences, 3,072 batched tokens, async off
- GPU memory utilization: 0.95
- CUDA graphs: graph ceiling 32; effective runtime capture sizes are recorded
in
snapshot/effective-process-args.txtand the server log - PCIe experimental paths: disabled
The exact host-bound compose file is
docker-compose.exact-popos.yml. It preserves the validated GPU UUID order.
Use docker-compose.portable.yml for a similar four-GPU machine; it maps the
four GPUs as CUDA devices 0 through 3.
Prerequisites
- Pop!_OS or Ubuntu with a working NVIDIA driver, Docker Engine, NVIDIA
Container Toolkit, Docker Compose v2,
curl, Python 3,ipmitool, and systemd. - Exactly four visible NVIDIA GPUs.
- The EXL3 model at
/mnt/llm_models/GLM-5.2-EXL3-TR3-3.0bpw, or setGLM52_MODEL_DIR. - A writable cache directory at
/mnt/llm_models/cache_glm52_exl3_text, or setGLM52_CACHE_DIR. - Python 3 with the
venvmodule. The installer creates an isolated/opt/glm52-hostenvironment and installs the hash-pinned NVML binding. - The raw CPU fan command is board-specific and was validated on the ASUS Pro WS WRX90E-SAGE SE. Do not install that service on another board until its IPMI command and readback format are verified.
On
Description
Reproducible GLM-5.2 EXL3 MTP3 serving and thermal profile for 4x RTX PRO 6000 Blackwell
Related repositories
Similar projects matched by category, topics, and programming language.
gavamedia
Featured
deltafin
Deltafin is a research project that runs the 2.8-trillion-parameter Mixture-of-Experts model Kimi K3 on a single Apple Silicon Mac (e.g., M1 Max with 64 GB) at about 16 seconds per token, using exact, reproducible inference with local or streaming expert loading.
jamesob
Featured
local-llm
A comprehensive guide for building and configuring a high-end local machine to run state-of-the-art LLMs, with detailed hardware choices, BIOS tuning, and Docker-based model serving.