jamesob
精选
local-llm
A comprehensive guide for building and configuring a high-end local machine to run state-of-the-art LLMs, with detailed hardware choices, BIOS tuning, and Docker-based model serving.
AI 仓库情报摘要
FR-AI / ANALYSIS为什么值得关注
它不仅采用现代开源模型的设计思路，还从零完成了模型、分词器、数据管线、训练、推理、聊天与测试的完整实现，配有文档和单元测试，适合深入理解 LLM 的各个模块。
适合谁使用
- 希望深入理解 Transformer 内部实现的工程师和研究者
- 想从零训练小型聊天模型的开发者和爱好者
- 需要可定制轻量级聊天助手的项目团队
- 学习 LLM 架构与训练流程的学生
典型使用场景
- 在合规语料上训练自己的小型聊天模型
- 开发多轮对话、问答、摘要、概念解释和创意写作应用
- 实现基础代码补全功能
- 作为学习现代 LLM 架构和训练流程的参考实现
项目优势
- 从零实现了 RoPE、RMSNorm、SwiGLU、GQA、权重绑定和 KV cache 等现代组件
- 覆盖模型、分词器、数据、训练、采样、聊天等环节的完整测试
- 包含分词器训练、数据准备、模型训练、聊天和评估的完整工具链
- 提供详细设计文档和可快速验证的 CPU 端到端示例
使用前须知
- MoE、滑动窗口注意力、Flash Attention 等仅为 NotImplementedError 钩子，并未实际实现
- 不附带预训练权重和数据集，需要用户自行训练分词器和模型
- 规模较轻量（约 1.23 亿参数），未包含分布式训练、量化、投机解码、LoRA、多模态等高级能力
README 快速开始
Zen-2
Zen-2 is an original, from-scratch, decoder-only transformer language model (~123M parameters) built for the workflow of a lightweight chat assistant: instruction following, Q&A, multi-turn conversation, summarization, concept explanation, basic code completion, and creative writing.
It is not a GPT-2 clone. The architecture follows the design choices of modern open-weight LLMs (LLaMA/Mistral/Qwen-style models):
- RoPE (Rotary Position Embeddings) instead of learned absolute positions
- RMSNorm instead of LayerNorm
- SwiGLU feed-forward blocks instead of GELU/ReLU MLPs
- Grouped Query Attention (GQA) for a smaller KV-cache memory footprint
- Weight-tied input embeddings / output LM head
- KV caching for efficient incremental decoding
- Byte-level BPE tokenizer (~32K vocab, trained from scratch)
Every module is independently implemented, documented, and unit-tested. See
docs/architecture.md for the full design writeup, including the math
intuition, complexity analysis, advantages, and limitations of each
component.
Project layout
Zen-2/
├── configs/ model + training YAML configs
├── zen2/
│ ├── config.py ModelConfig / TrainConfig dataclasses
│ ├── model/ RMSNorm, RoPE, GQA attention, SwiGLU, transformer, KV cache
│ ├── tokenizer/ byte-level BPE tokenizer
│ ├── data/ cleaning/filtering/dedup pipeline, binary dataset packer/loader
│ ├── training/ Trainer, optimizer, cosine scheduler, checkpoint manager
│ ├── inference/ sampling strategies, generation engine (KV-cache + streaming)
│ ├── chat/ multi-turn chat engine
│ ├── evaluation/ perplexity/throughput metrics, quality benchmark harness
│ └── utils/ seeding, logging
├── scripts/ CLI entry points (see below)
├── tests/ pytest unit + integration tests
├── docs/ design docs (architecture, training, tokenizer, inference, data)
├── examples/ quickstart walkthrough
└── weights/, logs/, datasets/ run outputs (empty by default)
Quickstart
pip install -r requirements.txt
# 1. Train a tokenizer on your corpus
python scripts/train_tokenizer.py --files my_corpus/*.txt --vocab-size 32000 \
--out tokenizer/zen2_tokenizer.json
# 2. Clean, tokenize, and pack your training
项目描述
A modern 124M-parameter decoder-only LLM built from scratch. RoPE · RMSNorm · SwiGLU · GQA · KV Cache · BPE Tokenizer · Full training pipeline. Research-grade, production-quality.
相关仓库与替代方案
根据分类、Topic 和编程语言匹配的相似项目。
slvDev
精选
esp32-ai
A 28.9 million parameter language model runs on an $8 ESP32-S3 microcontroller entirely on-device, generating simple stories at about 9.5 tokens per second.
gavamedia
精选
deltafin
Deltafin is a research project that runs the 2.8-trillion-parameter Mixture-of-Experts model Kimi K3 on a single Apple Silicon Mac (e.g., M1 Max with 64 GB) at about 16 seconds per token, using exact, reproducible inference with local or streaming expert loading.