0xwilliamortiz
Featured
openclaude-improved
OpenClaude is an open-source CLI coding agent that runs on any platform and supports a wide range of LLM providers, offering the same tools and workflows as Claude Code.
AI repository intelligence
FR-AI / ANALYSISWhy it is worth attention
It provides an extremely detailed, real-world build log for a 4x RTX PRO 6000 system with custom PCIe switches, achieving near line-rate P2P GPU communication while saving costs by using last-gen DDR4/PCIe4 hardware.
Who it is for
- AI researchers and engineers needing local SOTA inference
- Privacy-conscious users who avoid cloud AI services
- Hardware enthusiasts with budgets of $2k–$40k
- Developers wanting turnkey Docker configurations for large models
Use cases
- Running models like GLM-5.2-594B locally for agentic workloads
- Performing speech-to-text with cohere-transcribe in a privacy-respecting manner
- Building a local code assistant (opencode) with tooling and issue tracking
- Benchmarking GPU peer-to-peer bandwidth and latency
Strengths
- Itemized bill of materials with prices and sources (eBay, c-payne.com)
- Step-by-step BIOS and kernel parameter fixes for PCIe switches and multi-GPU P2P
- Proven performance: 27.5 GB/s unidirectional / 50.4 GB/s bidirectional P2P at sub-µs latency
- Cost-effective design: $5.6k base system (last-gen EPYC + DDR4) vs. $46k GPUs
Considerations
- Very high total cost (~$51k for the 4× RTX PRO 6000 build)
- Requires advanced hardware tinkering (carpentry, PCIe switch configuration, BIOS hacks)
- Limited to NVIDIA GPUs and requires specific kernel parameters (iommu=off) that may interfere with other workloads
README quick start
jamesob's guide to running SOTA LLMs locally
Note: nothing in this README aside from the tables was written by AI.
Have $2k burning a hole in your pocket and want some local, state-of-the-art machine intelligence?
How about $40k?
If Dario and Altman are giving you heartburn (they should be), read on to figure out how to run this new kind of computing locally.
In this repo you'll find
- the hardware I use to run SOTA locally,
- why I bought what and little-known secrets for configuring it,
- how I run speech-to-text (STT) locally,
- ready-to-run configuration for running models I think are good within Docker containers.
Contents
| Section | TL;DR |
|---|---|
| How much are you willing to spend? | $2k gets you Qwen and good STT (pretty far!); $40k gets you almost-Opus |
| Base system | Last-gen EPYC + eBay DDR4 for $5.6k |
| GPUs | 4× RTX PRO 6000, 384GB VRAM (where the money went) |
| c-payne switch sub-BOM | Indie PCIe switching from c-payne.com so GPUs talk peer-to-peer |
| GPU mount | A day of carpentry |
| Making the switch behave | BIOS bifurcation, link speed, ASPM |
| Kernel / GRUB params | iommu=off or NCCL hangs |
| ACS disable | Keep P2P traffic inside the switch fabric |
| GPU power limiting | Running $46k of silicon on a 110V circuit |
| Result | Gen4 line rate: 27.5/50.4 GB/s, sub-µs latency |
runners/ | Ready-to-run serving configs: GLM-5.2-594B: vLLM docker-compose, DCP4+MTP5, ~80 t/s @ 460k ctx |
runners/stt | Ready-to-run speech-to-text config with cohere-transcribe |
tools/ | measure-gpu-speed.sh: P2P bandwidth/latency benchmark |
| Resources | rtx6kpro repo, c-payne |
My setup
I was lucky/dumb enough to buy 4x RTX Pro 6000s back when they were cheaper. Because RAM is now so expensive, I opted to build a last-gen DDR4 system to host these cards, the parts for which I got off eBay. This allowed me to keep base system cost reasonable while still getting a lot of VRAM.
Another somewhat un
Description
Everything I know about running LLMs locally
Related repositories
Similar projects matched by category, topics, and programming language.
MoonshotAI
Featured
Kimi-K3
Kimi K3 is an open-weight, 2.8T-parameter native multimodal agentic model with a 1M-token context window, designed for frontier coding, knowledge work, and reasoning tasks.
slvDev
Featured
esp32-ai
A 28.9 million parameter language model runs on an $8 ESP32-S3 microcontroller entirely on-device, generating simple stories at about 9.5 tokens per second.