jamesob
Featured
local-llm
A comprehensive guide for building and configuring a high-end local machine to run state-of-the-art LLMs, with detailed hardware choices, BIOS tuning, and Docker-based model serving.
AI repository intelligence
FR-AI / ANALYSISWhy it is worth attention
Provides a turnkey, production-ready wrapper for a very large MoE model (35B active, 3B total parameters) with multi-modal vision, multi-token prediction speculative decoding, and 256K context, handling binary detection, auto-download, health checks, and graceful shutdown.
Who it is for
- Developers deploying local LLMs on high-VRAM NVIDIA GPUs (96–128 GB)
- Researchers and engineers using Qwen3.6-35B-A3B for long-context or multi-modal tasks
- NVIDIA DGX Spark (GB10) users seeking a convenient llama.cpp setup
- llama.cpp enthusiasts who need a reliable server launcher for heavy models
Use cases
- Local inference with OpenAI-compatible API for chat and completions
- Multi-modal Qwen-VL reasoning with vision projector and 256K context
- Speculative decoding with MTP heads for faster generation
- Long-document analysis or agentic workflows requiring persistent background serving
Strengths
- Automatic detection of llama-server binary and graceful failure fallbacks
- Health-check polling with spinner and elapsed time before declaring ready
- Automatic model download from Hugging Face with resume support and token handling
- Companion stop.sh with SIGTERM → SIGKILL escalation and PID file management
Considerations
- Requires 96–128 GB VRAM and a recent CUDA build of llama.cpp (no CPU inference)
- Linux only (bash, nohup, pgrep) – WSL works but native Windows is not supported
- Large download (~40 GB) and heavy memory usage; out-of-the-box config may not fit smaller GPUs without manual tuning
README quick start
Quick Start
Description
llama-server start/stop scripts for Qwen3.6-35B-A3B UD-Q8_K_XL GGUF on DGX Spark
Related repositories
Similar projects matched by category, topics, and programming language.
bjan
Featured
claudemux
Claudemux is a lightweight, dependency-minimal message bus that lets multiple Claude Code sessions communicate in real-time over tmux, enabling one session to ask another a question and receive the answer without manual intervention.
AARomanov1985
Audio-Cassette-Simulation
A set of bash scripts using ffmpeg to simulate vintage cassette tape audio profiles with noise, wow/flutter, bandwidth limits, and EQ.