An MLX port of Moonshot's 2.78T-parameter multimodal MoE Kimi-K3, featuring 1M-token context and novel attention mechanisms, but requiring over 870GB of memory—currently impossible to run on any existing Apple Silicon Mac.
27
No data
1
0
No data
2026-07-28
This repository demonstrates the most extensive porting of a frontier-scale MoE model to MLX, including bit-exact MXFP4 conversion, novel attention residual architecture, a fully validated vision tower, and REAP expert pruning—all documented with rigorous testing despite the model being too large to run on current hardware.
MLX (Apple Silicon) port of moonshotai/Kimi-K3 — Moonshot's 2.78T-total / 104B-active native-multimodal MoE, built on Kimi Delta Attention (KDA) and Attention Residuals (AttnRes), with a 1M-token context.
kimi_k3.py is a single, stock-mlx-lm-compatible model definition for the text
tower; kimi_k3_vision.py is the vision tower and projector; kimi_k3_vl/ is
the mlx-vlm wrapper that joins them.
scripts/convert.py is a streaming converter — mlx_lm convert cannot be used
here, because it materialises the whole model and K3 is 5.6 TB in bf16.
Read this before planning a deployment. Nothing in this repo runs on a single Mac. The smallest tier is ~870 GB against a 512 GB ceiling on the largest Apple Silicon machine that exists. See Reality check.
| Feature | Detail |
|---|---|
| MoE | 896 routed experts, top-16, moe_intermediate_size=3072; sigmoid router + aux-loss-free e_score_correction_bias; 2 shared experts on every token |
| Stable LatentMoE | routed experts run in a 3584-d latent, not the 7168-d residual: down_proj → experts → RMSNorm → up_proj |
| Attention | 93 layers: 69 KDA + 24 gated MLA, 96 heads, head_dim=128 |
| KDA | short conv (k=4) on q/k/v, low-rank forget gate (f_a/f_b), full-rank output gate, gate_lower_bound=-5.0 |
| MLA | q-LoRA rank 1536, kv-LoRA rank 512, NoPE (no rotary at all), sigmoid output gate (g_proj) |
| AttnRes | every 12 layers pushes the residual onto a stack; each sub-layer input is a softmax mixture over that stack, scored by a rank-1 [1, 7168] direction |
| Activation | SiTU-GLU: β·tanh(g/β)·σ(g) · β_lin·tanh(u/β_lin), β=4, β_lin=25 |
| Dense layer | layer 0 only (first_k_dense_replace=1, intermediate_size=33792) |
| Vocab | 163,840 · tiktoken · tie_word_embeddings=False |
| Source format | routed experts ship as MXFP4 (weight_packed u8 + e8m0 weight_scale, group 32); everything else bf16 |
Parameter budget — 97.94% of the model is routed experts:
| component | params | share |
|---|---|---|
| routed experts (MXFP4 in source) | 2722.7 B | 97.94% |
| KDA attention ×69 | 30.6 B | 1.10% |
| shared experts | 12.2 B | 0.44% |
| MLA attention ×24 | 5.6 B | 0.20% |
| latent-MoE up/down | 4.7 B | 0.17% |
| embed + lm_head |
MLX port of moonshotai/Kimi-K3 (2.78T multimodal MoE): streaming converter, REAP expert pruning, and per-language expert-overlap analysis
Similar projects matched by category, topics, and programming language.
Harness Engineering is a methodology for improving coding agent outputs by carefully crafting the environment around them—providing curated context, tools, and executable constraints that encode an organization’s nonfunctional requirements and cumulative lessons.
A 28.9 million parameter language model runs on an $8 ESP32-S3 microcontroller entirely on-device, generating simple stories at about 9.5 tokens per second.
mimic captures traffic from any iOS or web app and automatically generates a Python client library that lets you call the app's API like a regular library.