PipeNetwork
kimi-k3-mlx
An MLX port of Moonshot's 2.78T-parameter multimodal MoE Kimi-K3, featuring 1M-token context and novel attention mechanisms, but requiring over 870GB of memory—currently impossible to run on any existing Apple Silicon Mac.
AI repository intelligence
FR-AI / ANALYSISWhy it is worth attention
It demonstrates the first known trillion-scale model streaming from disk on a consumer machine, achieving 0.32 tokens/second with no cloud dependency and validated accuracy.
Who it is for
- Developers building offline or privacy-preserving AI applications
- Researchers exploring extreme model compression and streaming
- Hobbyists who want to run frontier models on personal hardware
- Engineers evaluating edge inference for large MoE architectures
Use cases
- Running Kimi K3 (2.78T) on a laptop for local reasoning without network
- Testing and prototyping large MoE models on affordable hardware
- Serving a 48B model (Kimi-Linear) at ~9 tok/s for lightweight tasks
- Hosting an OpenAI-compatible server that executes on the same machine
Strengths
- Self-contained C library with zero runtime dependencies (no BLAS, CUDA, Python)
- Validated against PyTorch with logits agreeing to 3.6e-06 and vision tower to 2.3e-06
- Efficient I/O: one pread per expert, cache-bypassing, and 3-bit VQ encoding for dense expert storage
- Absorbed KV cache cuts memory from 11.25 GB to 0.21 GB at 4K context
Considerations
- Very slow decode: 0.32–0.34 tok/s on K3 (over 50 seconds per short sentence)
- Requires large disk space (982 GB for K3) and internal NVMe for acceptable I/O
- Narrow usable RAM window: optimal performance only near 46 GB on a 64 GB system, with risk of paging beyond that
README quick start
WASTE — Weight-Aware Streaming Tensor Engine
Kimi K3 — 2.78 trillion parameters — running on a consumer laptop.
$ waste run ~/models/k3.waste 'What is the capital of Italy?'
waste: no --budget, using 46.24 GB of 64.00 GB (expert cache 17.56 GB)
The capital of Italy is **Rome**.
[16 tokens, 49.31 s, 0.32 tok/s | experts 3357 hit / 20195 miss = 14%]
WASTE is an embeddable inference engine written in C, with no third-party runtime dependencies. It keeps the model trunk in memory, streams selected experts directly from disk, and uses the remaining RAM as a bounded expert cache.
Its current proof point is the complete open-weights Kimi K3 model: 2.78 trillion parameters, converted into a 982 GiB container and running on a 64 GB MacBook Pro at 0.32–0.34 tokens per second. This is not a distilled, pruned, or reduced variant.
| Model | Container | Minimum RAM | Tested speed |
|---|---|---|---|
| Kimi K3 2.78T | 982 GiB | 29.05 GiB | 0.32–0.34 tok/s |
| Kimi-Linear 48B | 19 GiB | 1.86 GiB | 8.92 tok/s |
WASTE was written for that one model and that one constraint: K3 does not fit in the RAM of current mainstream consumer systems. It is 1.42 TB as published and 982 GB after conversion. But a mixture of experts activates about 4% of itself per token, so almost all of that weight is idle at any instant — and idle weight does not need to be in memory, it needs to be reachable in time. WASTE keeps it on disk in a layout where one expert costs exactly one read, streams what each token actually needs, and spends every remaining byte of RAM on the part that repeats.
Where this stands
The engine is correct: every layer is validated against a PyTorch reference, the final logits agree to 3.6e-06, and the vision tower matches its own oracle to 2.3e-06. It is also slow — a third of a token per second, fifty seconds for the sentence above.
Both of those matter, and the second one should not be read as a disclaimer. We are not aware of another published demonstration of a model this size streaming from disk on a consumer machine: we found none for trillion-scale NVMe streaming, and the best-documented 671B-class recipes assume a server with a terabyte of DDR5. That is a report of what our search turned up rather than a survey — this repository carries no b
Description
Run the full 2.78-trillion-parameter Kimi K3 model beyond available RAM by streaming activated weights directly from NVMe. A dependency-free, embeddable C inference engine.
Related repositories
Similar projects matched by category, topics, and programming language.
filipj9
Hermes-Control-Deck
Hermes Control is a mobile-first PWA control deck that provides a unified event-driven interface for monitoring and operating Hermes WebUI and Codex CLI AI runtimes from a browser or phone.
peditx
tauri-skills
A collection of battle-tested, parallel-executing skills for building any type of cross-platform desktop application with Tauri 2, Rust, React/TypeScript, and optional Python sidecar.