prima.cpp is a distributed inference engine that runs large LLMs (30-70B) across heterogeneous everyday devices like laptops, desktops, and phones, achieving low memory pressure and substantial speedups over llama.cpp.
64
No data
9
1
MIT
2026-06-30
It enables running 70B-level models on a home cluster of cheap, heterogeneous devices with memory usage below 10%, achieving up to 15x speed improvement over llama.cpp through piped-ring parallelism, heterogeneity-aware workload distribution, and automatic device selection.
prima.cpp is a distributed implementation of llama.cpp that lets you run 70B-level LLMs on your everyday devices—💻 laptops, 🖥️ desktops, 📱 phones, and tablets (GPU or no GPU, it’s all good). With it, you can run QwQ-32B, Qwen 2.5-72B, Llama 3-70B, or DeepSeek R1 70B right from your local home cluster!
Worried about OOM or your device stucking? Never again! prima.cpp keeps its memory pressure below 10%, you can run very large models while enjoying Tiktok (if you don't mind the inference speed).
How about speed? Built upon llama.cpp, but it’s 15x faster! 🚀 On my poor devices, QwQ-32B generates 11 tokens per second, and Llama 3-70B generates 1.5 tokens per second. That's about the same speed as audiobook apps, from slow to fast speaking. We plan to power a Home Siri soon, then we can have private chats without privacy concerns.
prima.cpp vs llama.cpp on QwQ 32B:
https://github.com/user-attachments/assets/9fa3e57f-3f6b-49f3-800e-df9d758a60c6
prima.cpp vs llama.cpp on DeepSeek R1 70B:
https://github.com/user-attachments/assets/9549290e-a56f-46e1-9549-74250c1e0f7e
And, if your devices are more powerful, you could unlock even more possibilities, like running LLM agents right in your home! If you do, we’d love to hear about it, just share your cluster setup and token throughput with us!
Table 1: Home cluster configurations.
| D1 | D2 | D3 | D4 | |
|---|---|---|---|---|
| Device | Mac M1 | Laptop | Desktop | Mate40Pro |
| OS | MacOS (UMA) | Linux | Linux | Linux (on HarmonyOS) |
| CPU | Apple M1 | Intel i9 | Intel i9 | Kirin 9000 |
| CPU Cores | 8 | 8 | 16 | 8 |
| RAM (available) | 2.4 GiB | 4.1 GiB | 9.7 GiB | 1.9 GiB |
| Disk Read Speed | 0.72 GB/s | 2.98 GB/s | 3.17 GB/s | 1.37 GB/s |
| GPU Type | Apple Metal | 3070 | 2080TI | - |
| VRAM (available) | - | 8 GiB | 11 GiB | - |
Device D4 runs inside a Termux-simulate
[Official] prima.cpp: Fast 30-70B LLM inference on heterogeneous and everyday home devices
Similar projects matched by category, topics, and programming language.
A CGAL-free, MIT-licensed reimplementation of the Alpha Wrapping algorithm that takes a defective triangle soup and produces a watertight, manifold, outward-oriented mesh with controllable level of detail and tightness.
A proof-of-concept tool that patches a synchronous three-slot audio queue in GTA V Enhanced, boosting frame rates from ~260 to ~499 FPS by removing coarse wait delays.
This repository ports Command & Conquer Generals: Zero Hour to run natively on Apple Silicon Macs, iPhones, and iPads without emulation, adding touch controls and modern rendering.