nanoGPT-Seis is an educational repository that walks through the entire pretraining lifecycle of a 113M-parameter GPT model for earthquake science, from data crawling to inference, with detailed explanations and measured metrics.
21
No data
3
0
MIT
2026-07-13
It provides a complete, reproducible, and transparent pipeline for building a small domain-specific language model, demystifying every stage with practical design decisions and real measurements, making LLM pretraining accessible to learners.
Train a small GPT for earthquake science — the entire LLM lifecycle, from a blank folder to a talking model, explained block by block.
Crawl → Clean → Tokenize → Model → Train → Infer, on 2× NVIDIA A30 (48 GB).
Six free data sources → crawl → clean/dedup → 16k BPE → 113M GQA+RoPE decoder → 2-GPU DDP training → streaming inference. Each stage is a section below.
nanoGPT-Seis is a teaching repository. It is not trying to be a great earthquake model — it is trying to make every stage of pretraining a language model legible: where the data comes from, how it is cleaned and deduplicated, how a tokenizer is built, why the Transformer looks the way it does, how it is trained across two GPUs, and how it is served. Every design decision is explained, and every number (perplexity, VRAM, tokens) is one we actually measured on this hardware.
The corpus mixes earthquake / seismology text (open-access papers via Crossref+Unpaywall, arXiv/EarthArXiv preprints, the "Earthquake Insights" Substack) with general text (Wikipedia + FineWeb-Edu) for plain-language fluency — about 24% domain / 76% general. A focused corpus lets a ~100M-param model become genuinely fluent on a single node, so you can see the whole loop close in a day, not a month. (Why the general mix? See §1.)
Status: the pretraining lifecycle is complete — crawl through inference.
| value | |
|---|---|
| Corpus | 533,248 docs · 485.7M words · 822.7M training tokens (≈2.4:1 general:domain) |
| Model | 113M params — decoder-only, GQA + RoPE + RMSNorm + SwiGLU |
| Hardware | 2× NVIDIA A30 (24 GB each), bf16, DDP — also runs on a single RTX 3090/4090, or 12–16 GB with a smaller batch ([§7.5](#75-running-on-on |
Similar projects matched by category, topics, and programming language.
Harness Engineering is a methodology for improving coding agent outputs by carefully crafting the environment around them—providing curated context, tools, and executable constraints that encode an organization’s nonfunctional requirements and cumulative lessons.
A 28.9 million parameter language model runs on an $8 ESP32-S3 microcontroller entirely on-device, generating simple stories at about 9.5 tokens per second.
mimic captures traffic from any iOS or web app and automatically generates a Python client library that lets you call the app's API like a regular library.