FareedKhan-dev/kimi-k3-in-c

AI 仓库情报摘要

FR-AI / ANALYSIS

为什么值得关注

它通过流式路由专家和打包trunk，让2.78万亿参数的MoE模型在无GPU、仅8GB内存的CPU机器上运行，并在不同内存预算下保持逐字节完全相同的输出。

适合谁使用

研究超大MoE模型CPU推理的工程师
验证内存压缩/流式技术是否保持输出一致性的研究人员
对极简依赖C推理引擎感兴趣的开发者
拥有大容量NVMe但内存有限的本地部署爱好者

典型使用场景

在8GB内存的笔记本电脑上使用打包trunk运行Kimi K3基础模型生成文本
使用--dump-cache-trace和tools/sim_cache.py离线回放专家缓存轨迹，规划缓存大小
将C引擎的logits与PyTorch参考实现逐元素对比，验证正确性
使用--layers配合部分分片进行开发测试，无需下载完整1.56TB模型

项目优势

无BLAS、无框架、无GPU；整个引擎约176KB，仅依赖libm和OpenMP
通过打包trunk将内存需求变成可调旋钮；O_DIRECT实测比页缓存更快
make test无需checkpoint即可运行完整引擎与参考实现的严格一致性验证
文档包含实测数据：内存阶梯、专家缓存轨迹、存储带宽和性能数据

使用前须知

速度很慢：laptop预设约32.69秒/token，server预设约10.69秒/token
需要下载1.56TB checkpoint并预留约1.7TB存储空间；存储带宽决定整体吞吐
仅面向Linux x86-64和AVX2+FMA；且为基础模型，没有chat模板

README 快速开始

kimi-k3-in-c

A 2.78-trillion-parameter model. One CPU. 8 GB of RAM.

Kimi K3 inference in portable C99.No BLAS. No framework. No GPU.

2.78Tparameters 1.56 TBcheckpoint on disk 8.24 GBpeak RSS, measured 176 KBthe whole engine 0GPUs

$ ./bin/k3 ~/k3model --trunk ~/k3trunk --preset laptop \
           --tok ~/k3model --prompt "The capital of France is" --gen 8 --incremental

--- generated text ---
 Paris.",
+            "The Eiffel
----------------------
8 tokens in 261.5 s, 32.69 s/token average
PEAK RSS for the whole run: 8.24 GB

Slow, and answering correctly, in 8.24 GB, from a checkpoint of 1.56 TB. This is a base model, so what follows " Paris." is a continuation rather than a reply; there is no chat template. Give it more memory and the answer does not change, only the clock:

$ ./bin/k3 ~/k3model --trunk ~/k3trunk --preset server \
           --tok ~/k3model --prompt "def fibonacci(n):" --gen 28 --incremental

--- generated text ---
    if n  [prompt] [memory] [generation] [diagnostics]

`` is the directory holding the .safetensors shards. It is required for any run, but --help, --version and --list-presets work without it:

./bin/k3 --help
./bin/k3 --version
./bin/k3 --list-presets

Prompt options

Exactly one of these is required. Passing none, or more than one, is a usage error (exit 2).

flag	argument
`--prompt`	`TEXT`	tokenize TEXT and run it. Requires `--tok`.
`--prompt-file`	`PATH`	tokenize the file's bytes. Requires `--tok`. Preferred for anything non-ASCII: the shell re-encodes `argv`, whereas a file is read verbatim
`--ids`	`1,2,3`	token ids directly. No tokenizer is loaded at all, so this works on a machine with no tokenizer files. The reproducible channel the tests use

# text in
./bin/k3 ~/k3model --tok ~/k3model --prompt "The capital of France is" ...

# text in, from a file. Use this for CJK, emoji, accents
printf 'La capitale de la France est' > /tmp/p.txt
./bin/k3 ~/k3model --tok ~/k3model --prompt-file /tmp/p.txt ...

# ids in, ids out, no tokenizer needed
./bin/k3 ~/k3model --ids 1008,10484,318,15383,387 ...

Memory options

flag	argument	default
`--preset`	`NAME`	none	`laptop` · `desktop` · `workstation` · `server` · `max`. Sets both budgets below
`--trun

项目描述

A 2.78-trillion-parameter Kimi K3 running inference on a single CPU in 8.24 GB of RAM. Portable C99: no BLAS, no framework, no GPU.