0xwilliamortiz
精选
openclaude-improved
OpenClaude is an open-source CLI coding agent that runs on any platform and supports a wide range of LLM providers, offering the same tools and workflows as Claude Code.
AI 仓库情报摘要
FR-AI / ANALYSIS为什么值得关注
该仓库提供了4× RTX PRO 6000系统搭配定制PCIe交换机的真实构建日志，实现了接近线速的GPU点对点通信，同时通过使用上一代DDR4/PCIe4硬件节省成本。
适合谁使用
- 需要本地运行SOTA推理的AI研究人员与工程师
- 注重隐私、避免使用云端AI服务的用户
- 预算在2千至4万美元之间的硬件爱好者
- 希望获得开箱即用的大型模型Docker配置的开发者
典型使用场景
- 本地运行GLM-5.2-594B等模型用于代理型工作负载
- 使用cohere-transcribe进行隐私保护的语音转文字
- 构建本地代码助手（opencode），集成工具链与问题追踪
- 测试GPU点对点带宽与延迟性能
项目优势
- 详细的物料清单（含价格与来源：eBay、c-payne.com）
- 针对PCIe交换机与多GPU点对点的BIOS及内核参数逐步修复方案
- 已验证性能：单向27.5 GB/s、双向50.4 GB/s，亚微秒延迟
- 成本优化设计：基础系统仅$5,600（上一代EPYC+DDR4），GPU占$46,000
使用前须知
- 总成本极高（4× RTX PRO 6000配置约$51,000）
- 需要高级硬件调校（木工、PCIe交换机配置、BIOS修改）
- 仅支持NVIDIA GPU，且需特定内核参数（iommu=off），可能影响其他工作负载
README 快速开始
jamesob's guide to running SOTA LLMs locally
Note: nothing in this README aside from the tables was written by AI.
Have $2k burning a hole in your pocket and want some local, state-of-the-art machine intelligence?
How about $40k?
If Dario and Altman are giving you heartburn (they should be), read on to figure out how to run this new kind of computing locally.
In this repo you'll find
- the hardware I use to run SOTA locally,
- why I bought what and little-known secrets for configuring it,
- how I run speech-to-text (STT) locally,
- ready-to-run configuration for running models I think are good within Docker containers.
Contents
| Section | TL;DR |
|---|---|
| How much are you willing to spend? | $2k gets you Qwen and good STT (pretty far!); $40k gets you almost-Opus |
| Base system | Last-gen EPYC + eBay DDR4 for $5.6k |
| GPUs | 4× RTX PRO 6000, 384GB VRAM (where the money went) |
| c-payne switch sub-BOM | Indie PCIe switching from c-payne.com so GPUs talk peer-to-peer |
| GPU mount | A day of carpentry |
| Making the switch behave | BIOS bifurcation, link speed, ASPM |
| Kernel / GRUB params | iommu=off or NCCL hangs |
| ACS disable | Keep P2P traffic inside the switch fabric |
| GPU power limiting | Running $46k of silicon on a 110V circuit |
| Result | Gen4 line rate: 27.5/50.4 GB/s, sub-µs latency |
runners/ | Ready-to-run serving configs: GLM-5.2-594B: vLLM docker-compose, DCP4+MTP5, ~80 t/s @ 460k ctx |
runners/stt | Ready-to-run speech-to-text config with cohere-transcribe |
tools/ | measure-gpu-speed.sh: P2P bandwidth/latency benchmark |
| Resources | rtx6kpro repo, c-payne |
My setup
I was lucky/dumb enough to buy 4x RTX Pro 6000s back when they were cheaper. Because RAM is now so expensive, I opted to build a last-gen DDR4 system to host these cards, the parts for which I got off eBay. This allowed me to keep base system cost reasonable while still getting a lot of VRAM.
Another somewhat un
项目描述
Everything I know about running LLMs locally
相关仓库与替代方案
根据分类、Topic 和编程语言匹配的相似项目。
MoonshotAI
精选
Kimi-K3
Kimi K3 is an open-weight, 2.8T-parameter native multimodal agentic model with a 1M-token context window, designed for frontier coding, knowledge work, and reasoning tasks.
slvDev
精选
esp32-ai
A 28.9 million parameter language model runs on an $8 ESP32-S3 microcontroller entirely on-device, generating simple stories at about 9.5 tokens per second.