How LLMs Work

An interactive app that walks through every stage of the LLM pipeline — from pattern matching to training a transformer from scratch — with working code you can run locally.

• How LLMs Work
  • Stack
  • Setup
  • Sections
    • 1. Simple Chat — Pattern Matching
    • 2. XOR Neural Net — Backpropagation
    • 3. Basic Tokenizer — BPE From Scratch
    • 4. Train Embeddings — Word2Vec Skip-Gram
    • 5. Train Transformer — GPT From Scratch
  • Architecture
  • References

Stack

• Hono — server, JSX rendering, and client components
• Vite — dev server + bundling

Setup

Requires pnpm and Node.js 20+.

pnpm install
pnpm dev

If pnpm reports ignored build scripts, run:

pnpm approve-builds

Select esbuild with Space, press Enter, then rerun pnpm install.

Sections

1. Simple Chat — Pattern Matching

The simplest possible "AI": an ELIZA-style chatbot that responds with if-statements, streamed word by word via SSE. Same plumbing as ChatGPT, zero intelligence.

• Route: src/routes/simple-chat.ts
• Hook: src/client/hooks/use-simple-chat.ts

2. XOR Neural Net — Backpropagation

Trains a neural network live. A single-layer perceptron fails on XOR (proving Minsky/Papert right). A multi-layer network succeeds via backpropagation — the same algorithm every neural network uses today.

• Route: src/routes/neural-net/
• Hook: src/client/hooks/use-neural-net-chat.tsx
• Component: src/client/components/neural-net-result/

3. Basic Tokenizer — BPE From Scratch

A from-scratch BPE implementation that trains on your input text. Watch merge steps animate as the algorithm builds a vocabulary from characters to words.

• Route: src/routes/bpe-tokenize.ts
• Hook: src/client/hooks/use-bpe-tokenize-chat.tsx