MoonshotAI
Featured
Kimi-K3
Kimi K3 is an open-weight, 2.8T-parameter native multimodal agentic model with a 1M-token context window, designed for frontier coding, knowledge work, and reasoning tasks.
Qwen3.6-27B-NVFP4-DFlash-DGX-Spark
A production-ready vLLM deployment wrapper for the Qwen3.6-27B model with NVFP4 quantization and DFlash speculative decoding, providing a Docker container and scripts for easy server setup.
Stars
9
7-day growth
No data
Forks
0
Open issues
0
License
No data
Last updated
2026-07-30
AI repository intelligence
FR-AI / ANALYSISWhy it is worth attention
It packages a highly quantized 27B parameter model with speculative decoding and a 256K context window into a single, easy-to-deploy Docker solution optimized for ARM64 NVIDIA GPUs.
Who it is for
- Developers deploying large language models in production
- Researchers needing fast inference with long context
- Users with ARM64 NVIDIA GPUs (e.g., A100, RTX 6000)
- Teams requiring an OpenAI-compatible API for LLM integration
Use cases
- Building real-time chat applications with streaming
- Code generation and tool-calling via Qwen3-coder parser
- Analyzing long documents (up to 256K tokens) with deep-context throughput
- Multimodal tasks (image input) handled by the vision-capable model
Strengths
- Easy one-command deployment via Docker with start/stop scripts
- High throughput from DFlash speculative decoding (10 draft tokens) and Flash Attention
- Very long context window of up to 262,144 tokens
- Comprehensive custom Jinja chat template with thinking, tool use, and error recovery
Considerations
- Requires a specific ARM64 GPU with at least 40 GB VRAM (e.g., A100, RTX 6000)
- Currently configured for a maximum of 4 concurrent sequences, limiting high-traffic use
- Depends on Docker with NVIDIA Container Toolkit and CUDA 12.x, increasing setup complexity
README quick start
Qwen3.6-27B (NVFP4) DFlash for DGX Spark
A production-ready vLLM deployment wrapper for Qwen3.6-27B — an NVFP4-quantized dense model (~27B params) with DFlash speculative decoding.
This repo bundles a ready-to-run Docker container, a custom chat template, and start/stop scripts so you can spin up a fully OpenAI-compatible inference server in minutes.
✨ Key Features
| Feature | Details |
|---|---|
| Model | Qwen3.6-27B-NVFP4 — NVFP4 quantised dense (~27B params) |
| Vision | Supports image input (multimodal) |
| Quantization | ModelOpt (--quantization modelopt) |
| Inference Engine | vLLM (nightly aarch64) with Flash Attention backend |
| Speculative Decoding | DFlash, draft model z-lab/Qwen3.6-27B-DFlash, 10 speculative tokens |
| Context Window | Up to 262 144 tokens (256K) |
| OpenAI-Compatible API | /v1/chat/completions, /v1/completions, /v1/models |
| Served Model Name | qwen36-27b-nvidia-nvfp4-dflash |
| Tool Use | Qwen3-coder tool-call parser, auto tool choice enabled |
| Thinking/Reasoning | CoT / chain-of-thought with `` block support (configurable) |
| Reasoning Parser | Qwen3-specific parser via --reasoning-parser qwen3 |
| Streaming | Full SSE streaming support |
| Prefix Caching | Enabled via --enable-prefix-caching |
| Chunked Prefill | Enabled via --enable-chunked-prefill |
| KV Cache | bfloat16 (--kv-cache-dtype bfloat16) |
| Custom Chat Template | Full Jinja template with tool use and thinking support |
| ARM64 Native | vLLM nightly aarch64 image |
📊 Performance
Decode Benchmark (Concurrency Scaling)
256 tokens output · 1–4 concurrent streams · total 33.7 s
| Concurrency | TTFT | Streams | Aggregate (tok/s) | Stream (tok/s) |
|---|---|---|---|---|
| ×1 | 133 ms | 1/1 | 45.6 | 45.6 |
| ×2 | 566 ms | 2/2 | 71.7 | 37.1 |
| ×3 | 718 ms | 3/3 | 86.0 | 33.9 |
| ×4 | 771 ms | 4/4 | 102.4 | 29.8 |
Aggregate — total tok/s across all concurrent streams. Stream — per-stream average.
Deep-Context Throughput
Deep-context throughput benchmark (salted prompts to avoid prefix cache):
| Config | Val
Description
Qwen3.6-27B (NVFP4) DFlash for DGX Spark
Related repositories
Similar projects matched by category, topics, and programming language.
jamesob
Featured
local-llm
A comprehensive guide for building and configuring a high-end local machine to run state-of-the-art LLMs, with detailed hardware choices, BIOS tuning, and Docker-based model serving.
makecindy
Featured
cindy
Cindy is an open-source AI agent that runs locally on your machine, integrates multiple AI harnesses and models, and provides memory, skills, and automation to perform real work in your projects and apps.