NousResearch
Featured
hermes-agent
Hermes Agent is an open-source, self-improving AI agent with a built-in learning loop that creates skills from experience, runs across platforms, and supports any LLM provider.
Why it is worth attention
It packs 100x more parameters than previous microcontroller LLMs by storing most of the model in flash instead of RAM, using Google's Per-Layer Embeddings technique.
Who it is for
- Embedded systems developers
- AI researchers exploring efficient inference
- Hobbyists building tiny AI gadgets
- Engineers working on edge AI privacy solutions
Use cases
- On-device story generation without internet connectivity
- Low-cost AI demonstrations for education or prototyping
- Privacy-preserving text generation where data never leaves the device
Strengths
- Very low hardware cost ($8 chip) for a language model
- Fully on-device inference with no server dependency
- Novel memory hierarchy using flash lookup table to dramatically reduce RAM needs
Considerations
- Model only produces simple stories and cannot answer questions or follow instructions
- Limited reasoning capability due to small 'thinking' core in SRAM
- Requires custom firmware and wiring knowledge to replicate
README quick start
Running a 28.9M parameter LLM on an $8 microcontroller
Open to Work · 𝕏 slvDev · LinkedIn
This is a 28.9 million parameter language model that generates text on an ESP32-S3, a microcontroller that costs about $8. It runs on the chip itself, with nothing sent to a server, and it writes each word to a small screen wired to the chip at roughly 9 tokens per second. The last language model people ran on a chip like this had 260 thousand parameters, so this one holds about a hundred times more. It fits because most of the model lives in flash instead of RAM, using an idea from Google's Gemma models called Per-Layer Embeddings.
The numbers
| Parameters | 28.9M stored (25M of them in a flash lookup table) |
| Chip | ESP32-S3, about $8, with 512KB SRAM, 8MB PSRAM and 16MB flash |
| Speed | about 9.5 tok/s end to end (9.7 tok/s of pure compute) |
| Connectivity | none, everything runs on the device |
| Model size | 14.9MB at 4-bit |
Why it is hard, and how it fits anyway
A microcontroller has very little fast memory. The ESP32-S3 gives you 512KB of SRAM. Normally the whole model has to be reachable from there, which keeps you stuck with tiny models, and that is why the previous model on a chip like this had only 260 thousand parameters.
The way around it is to stop putting the model in fast memory at all. Most of a language model's parameters sit in an embedding table, which the model reads from rather than computes on. So you can leave that 25 million row table in slow flash and pull only the few rows each token needs, about 450 bytes, while the small part that does the actual work stays in fast memory. The large model then costs almost nothing to run, because you never load most of it. It just sits in flash and gets sampled a little at a time.
That idea is Google's Per-Layer Embeddings, from Gemma 3n and Gemma 4. Here it runs on the memory layout of a microcontroller instead of a phone or a GPU. As far as I can tell, nobody had tried it on a chip this small.
SRAM (fast, tiny) the "thinking" core, used on every token
PSRAM (medium) t
Related repositories
Similar projects matched by category, topics, and programming language.
littledivy
Featured
mimic
mimic captures traffic from any iOS or web app and automatically generates a Python client library that lets you call the app's API like a regular library.
joeseesun
Featured
qiaomu-goal-meta-skill
qiaomu-goal-meta-skill transforms vague requests into a copy-ready Codex /goal command with defaults, verification, boundaries, and pause conditions.