slvDev
Featured
esp32-ai
A 28.9 million parameter language model runs on an $8 ESP32-S3 microcontroller entirely on-device, generating simple stories at about 9.5 tokens per second.
smol-kimi-k3
A 49-million-parameter language model that implements key architectural ideas from Moonshot AI's Kimi K3 (including Kimi Delta Attention, Gated MLA, Stable LatentMoE, and per-head Muon) and can be trained from scratch on TinyStories using a single 8GB GPU.
Stars
46
7-day growth
No data
Forks
2
Open issues
0
License
No data
Last updated
2026-07-28
Why it is worth attention
It makes complex frontier-model components—like linear attention with delta rule, sparse MoE with quantile balancing, and block attention residuals—accessible for hands-on study and modification on consumer hardware, without requiring massive compute.
Who it is for
- Researchers and students exploring modern LLM architectures
- Practitioners wanting to experiment with MoE and linear attention on limited budget
- AI engineers interested in novel optimizers like Muon and Per-Head Muon
- Hobbyists who want to train a non‑standard Transformer from scratch at home
Use cases
- Understanding and iterating on Kimi K3-inspired components at small scale
- Training a language model on a desktop GPU for educational purposes
- Benchmarking training throughput and memory usage of hybrid KDA/MLA with MoE
- Testing autoregressive decoding with recurrent attention caches
Strengths
- Preserves the unusual hybrid attention and MoE design of a frontier model rather than replacing it with a standard Transformer
- Runs on an 8GB GPU with verified throughput (901 tok/s at batch 16) and peak VRAM under 4 GB
- Includes a live training dashboard and a full verification test suite (6 passing tests)
- Well-documented with architecture diagrams, model card, and measured benchmarks
Considerations
- Trained only on TinyStories, so generated text is limited to simple children's stories
- Omitted the vision tower (401M parameters) and does not claim to reproduce Kimi K3's capabilities
- First CUDA call requires ~75 seconds to compile Triton kernels, and the model is not optimized for production inference
README quick start
Quick start
Description
A 49M-parameter Kimi K3-inspired language model trainable on one 8GB GPU
Related repositories
Similar projects matched by category, topics, and programming language.
jamesob
Featured
local-llm
A comprehensive guide for building and configuring a high-end local machine to run state-of-the-art LLMs, with detailed hardware choices, BIOS tuning, and Docker-based model serving.
makecindy
Featured
cindy
Cindy is an open-source AI agent that runs locally on your machine, integrates multiple AI harnesses and models, and provides memory, skills, and automation to perform real work in your projects and apps.