Why it is worth attention

It implements the full LLM stack from scratch rather than wrapping existing model code, including RoPE, RMSNorm, SwiGLU, GQA, weight tying, KV caching, a custom BPE tokenizer, data pipeline, training, inference, chat, and tests.

Who it is for

LLM engineers and developers learning transformer internals
Researchers or hobbyists building small models from scratch
Developers needing a customizable lightweight chat assistant base
NLP students wanting a documented reference implementation

Use cases

Training a custom small chat model on license-permitted corpora
Building multi-turn chat, Q&A, summarization, and creative writing applications
Basic code completion and concept explanation
Educational exploration of modern LLM architecture and training pipelines

Strengths

Modern architecture from scratch: RoPE, RMSNorm, SwiGLU, GQA, weight tying, and KV cache
Comprehensive tests cover model, tokenizer, data, training, sampling, and chat
Full pipeline included: tokenizer training, data preparation, training, chat, and evaluation
Detailed docs and a CPU-friendly end-to-end quickstart example

Considerations

MoE, sliding-window attention, Flash Attention, and other advanced features exist only as NotImplementedError hooks, not working features
No pretrained weights or datasets are bundled; users must train their own tokenizer and model
Scale is lightweight (~123M parameters), with no distributed training, quantization, speculative decoding, LoRA, or multimodal support

README quick start

Zen-2

Zen-2 is an original, from-scratch, decoder-only transformer language model (~123M parameters) built for the workflow of a lightweight chat assistant: instruction following, Q&A, multi-turn conversation, summarization, concept explanation, basic code completion, and creative writing.

It is not a GPT-2 clone. The architecture follows the design choices of modern open-weight LLMs (LLaMA/Mistral/Qwen-style models):

RoPE (Rotary Position Embeddings) instead of learned absolute positions
RMSNorm instead of LayerNorm
SwiGLU feed-forward blocks instead of GELU/ReLU MLPs
Grouped Query Attention (GQA) for a smaller KV-cache memory footprint
Weight-tied input embeddings / output LM head
KV caching for efficient incremental decoding
Byte-level BPE tokenizer (~32K vocab, trained from scratch)

Every module is independently implemented, documented, and unit-tested. See docs/architecture.md for the full design writeup, including the math intuition, complexity analysis, advantages, and limitations of each component.

Project layout

Zen-2/
├── configs/            model + training YAML configs
├── zen2/
│   ├── config.py        ModelConfig / TrainConfig dataclasses
│   ├── model/            RMSNorm, RoPE, GQA attention, SwiGLU, transformer, KV cache
│   ├── tokenizer/        byte-level BPE tokenizer
│   ├── data/              cleaning/filtering/dedup pipeline, binary dataset packer/loader
│   ├── training/          Trainer, optimizer, cosine scheduler, checkpoint manager
│   ├── inference/         sampling strategies, generation engine (KV-cache + streaming)
│   ├── chat/               multi-turn chat engine
│   ├── evaluation/         perplexity/throughput metrics, quality benchmark harness
│   └── utils/               seeding, logging
├── scripts/              CLI entry points (see below)
├── tests/                pytest unit + integration tests
├── docs/                  design docs (architecture, training, tokenizer, inference, data)
├── examples/              quickstart walkthrough
└── weights/, logs/, datasets/   run outputs (empty by default)

Quickstart

pip install -r requirements.txt

# 1. Train a tokenizer on your corpus
python scripts/train_tokenizer.py --files my_corpus/*.txt --vocab-size 32000 \
    --out tokenizer/zen2_tokenizer.json

# 2. Clean, tokenize, and pack your training

Description

A modern 124M-parameter decoder-only LLM built from scratch. RoPE · RMSNorm · SwiGLU · GQA · KV Cache · BPE Tokenizer · Full training pipeline. Research-grade, production-quality.

Related repositories

Similar projects matched by category, topics, and programming language.

jamesob

Featured

local-llm

A comprehensive guide for building and configuring a high-end local machine to run state-of-the-art LLMs, with detailed hardware choices, BIOS tuning, and Docker-based model serving.

AI & Machine LearningLarge Language Models

1,660

slvDev

Featured

esp32-ai

A 28.9 million parameter language model runs on an $8 ESP32-S3 microcontroller entirely on-device, generating simple stories at about 9.5 tokens per second.

AI & Machine LearningLarge Language Models

1,960

gavamedia

Featured

deltafin

Deltafin is a research project that runs the 2.8-trillion-parameter Mixture-of-Experts model Kimi K3 on a single Apple Silicon Mac (e.g., M1 Max with 64 GB) at about 16 seconds per token, using exact, reproducible inference with local or streaming expert loading.

AI & Machine LearningLarge Language Models

304