为什么值得关注

它首次展示了在消费级设备上从磁盘流式加载万亿级模型，以 0.32 tokens/秒的速度运行，无需云端依赖，且精度经过验证。

适合谁使用

构建离线或隐私保护 AI 应用的开发者
研究极端模型压缩与流式加载的研究人员
希望在个人硬件上运行前沿模型的爱好者
评估大型 MoE 架构边缘推理的工程师

典型使用场景

在笔记本电脑上本地运行 Kimi K3（2.78T）进行推理，无需网络
在廉价硬件上测试和原型化大型 MoE 模型
以约 9 tok/s 的速度运行 48B 模型（Kimi-Linear）处理轻量任务
在同一台机器上托管兼容 OpenAI 的推理服务器

项目优势

自包含 C 库，零运行时依赖（无 BLAS、CUDA、Python）
与 PyTorch 验证：最终 logits 精度 3.6e-06，视觉塔精度 2.3e-06
高效 I/O：每个专家仅一次 pread，绕过缓存，使用 3 位 VQ 编码压缩专家权重
吸收式 KV 缓存将内存从 11.25 GB 降至 0.21 GB（4K 上下文）

使用前须知

K3 解码速度极慢：0.32–0.34 tok/s（短句耗时超过 50 秒）
需要大量磁盘空间（K3 需 982 GB）且必须使用内部 NVMe 才能获得可接受的 I/O
可用内存窗口狭窄：64 GB 系统在约 46 GB 时性能最佳，超出后可能触发换页

README 快速开始

WASTE — Weight-Aware Streaming Tensor Engine

Kimi K3 — 2.78 trillion parameters — running on a consumer laptop.

$ waste run ~/models/k3.waste 'What is the capital of Italy?'
waste: no --budget, using 46.24 GB of 64.00 GB (expert cache 17.56 GB)
The capital of Italy is **Rome**.
[16 tokens, 49.31 s, 0.32 tok/s | experts 3357 hit / 20195 miss = 14%]

WASTE is an embeddable inference engine written in C, with no third-party runtime dependencies. It keeps the model trunk in memory, streams selected experts directly from disk, and uses the remaining RAM as a bounded expert cache.

Its current proof point is the complete open-weights Kimi K3 model: 2.78 trillion parameters, converted into a 982 GiB container and running on a 64 GB MacBook Pro at 0.32–0.34 tokens per second. This is not a distilled, pruned, or reduced variant.

Model	Container	Minimum RAM	Tested speed
Kimi K3 2.78T	982 GiB	29.05 GiB	0.32–0.34 tok/s
Kimi-Linear 48B	19 GiB	1.86 GiB	8.92 tok/s

WASTE was written for that one model and that one constraint: K3 does not fit in the RAM of current mainstream consumer systems. It is 1.42 TB as published and 982 GB after conversion. But a mixture of experts activates about 4% of itself per token, so almost all of that weight is idle at any instant — and idle weight does not need to be in memory, it needs to be reachable in time. WASTE keeps it on disk in a layout where one expert costs exactly one read, streams what each token actually needs, and spends every remaining byte of RAM on the part that repeats.

Where this stands

The engine is correct: every layer is validated against a PyTorch reference, the final logits agree to 3.6e-06, and the vision tower matches its own oracle to 2.3e-06. It is also slow — a third of a token per second, fifty seconds for the sentence above.

Both of those matter, and the second one should not be read as a disclaimer. We are not aware of another published demonstration of a model this size streaming from disk on a consumer machine: we found none for trillion-scale NVMe streaming, and the best-documented 671B-class recipes assume a server with a terabyte of DDR5. That is a report of what our search turned up rather than a survey — this repository carries no b

项目描述

Run the full 2.78-trillion-parameter Kimi K3 model beyond available RAM by streaming activated weights directly from NVMe. A dependency-free, embeddable C inference engine.

waste