jamesob
Featured
local-llm
A comprehensive guide for building and configuring a high-end local machine to run state-of-the-art LLMs, with detailed hardware choices, BIOS tuning, and Docker-based model serving.
Unsloth-Qwen3.6-35b-NVFP4-DGX-Spark
This repository provides a vLLM-based deployment for running the unsloth/Qwen3.6-35B-A3B-NVFP4 MoE model on NVIDIA DGX Spark (GB10) with optimized FP8/NVFP4 inference, supporting up to 256K context, multi-modal input, and speculative decoding.
Stars
28
7-day growth
No data
Forks
4
Open issues
0
License
No data
Last updated
2026-07-11
AI repository intelligence
FR-AI / ANALYSISWhy it is worth attention
It demonstrates production-ready deployment of a large Mixture-of-Experts model (35B total, 3B active) on a single consumer-grade GPU (GB10) with cutting-edge optimizations like FlashInfer B12X linear kernels, FP8 KV cache, chunked prefill, and MTP speculative decoding, achieving ~80 tok/s decode speed and low latency.
Who it is for
- NVIDIA DGX Spark / GB10 users
- Developers deploying large MoE models locally
- Researchers evaluating Qwen3.6 performance on edge hardware
- vLLM enthusiasts interested in NVFP4 optimizations
Use cases
- Local coding assistant with tool use and reasoning
- Multi-modal chat with image inputs (up to 4 per request)
- High-context document analysis (up to 262K tokens)
- Prototyping and testing Qwen3.6 model capabilities on a single GPU
Strengths
- Optimized specifically for SM121 (GB10) with native NVFP4 and FlashInfer kernels
- Speculative decoding with ~86% acceptance rate boosts throughput
- Low TTFT: P50 103ms, P95 107ms for 200-token prompt
- Supports vision, tool use, and Qwen3 CoT reasoning out of the box
Considerations
- Requires specific NVIDIA GPU with compute capability 12.1 (GB10/DGX Spark), not universal
- First startup takes 10–20 minutes due to torch.compile and weight caching
- Limited concurrent sequences (24) and batched tokens (32768) due to memory constraints
README quick start
Quick Start
Description
vLLM deployment for Unsloth Qwen3.6-35B-A3B-NVFP4-Fast on NVIDIA DGX Spark
Related repositories
Similar projects matched by category, topics, and programming language.
bjan
Featured
claudemux
Claudemux is a lightweight, dependency-minimal message bus that lets multiple Claude Code sessions communicate in real-time over tmux, enabling one session to ask another a question and receive the answer without manual intervention.
AARomanov1985
Audio-Cassette-Simulation
A set of bash scripts using ffmpeg to simulate vintage cassette tape audio profiles with noise, wow/flutter, bandwidth limits, and EQ.