TanStack
Featured
router
TanStack Router is a type-safe, data-driven React router with built-in caching, prefetching, and nested layouts, while TanStack Start extends it into a full-stack SSR framework.
basically-agent-evals
A hands-on TypeScript curriculum that teaches how to build LLM evaluations by exposing common pitfalls like LLM-as-judge biases, with progressive runnable examples from simple assertions to a live diagnostic console.
Stars
13
7-day growth
No data
Forks
0
Open issues
0
License
No data
Last updated
2026-07-13
AI repository intelligence
FR-AI / ANALYSISWhy it is worth attention
It tackles the widespread problem of unreliable LLM evaluations by providing reproducible demonstrations of biases (position, length, sycophancy), concrete fixes (rubrics, calibration, probes), and a complete progression from foundational tests to a real RAG system debugger, all runnable offline with mock mode.
Who it is for
- LLM engineers and developers building evaluation pipelines
- AI/ML researchers studying evaluation reliability
- Technical product managers overseeing LLM deployment
- Educators teaching LLM evaluation best practices
Use cases
- Debugging and attributing failures in RAG systems (retrieval vs. generation)
- Setting up regression gates in CI/CD to catch LLM output regressions
- Comparing prompt variations or model choices with a calibrated evaluation framework
- Teaching or learning the principles of robust LLM evaluation through hands-on exercises
Strengths
- Covers seven specific LLM-as-judge biases with reproducible mock-mode examples
- Provides both deterministic probes and rubric-based judges for granular failure analysis
- Includes calibration against human agreement and a live console for interactive debugging
- Requires no build step, runs on Node.js 18+ with optional offline mock mode
Considerations
- Assumes familiarity with TypeScript and Node.js environment
- The mock mode uses canned responses, which may not cover all edge cases
- Full functionality requires an OpenRouter API key for live LLM calls
README quick start
Your Evals Are Lying To You
A minimal, hands-on TypeScript curriculum for LLM evaluations — how to build them, and how they lie to you. Built for a 60-minute talk of the same name. Each folder is self-contained and runnable, progressing from a five-line assertion to a live diagnostic console pointed at a real RAG system.
This repo is a teaching tool. In the taxonomy of the word "harness," it's the older, sharper meaning — the eval harness: measure a model against known answers and get back a score you can trust. (Its sibling, the agent harness — give a model tools and a task — lives at basically-ai-harness.)
What is an eval?
Start from what you already know — a unit test:
expect(add(2, 2)).toBe(4);
An eval is that same test, for software that breaks the three things a unit test relies on:
| A unit test assumes… | …an LLM breaks it because… | …so an eval needs |
|---|---|---|
| one fixed input | there are endless phrasings of the same request | a dataset of inputs, not one |
| one exact right answer | "good" is fuzzy — tone, helpfulness, "did it answer?" | a notion of good (a property, rubric, or reference) |
exact match (===) | output varies run to run | a tolerant checker, not string equality |
The clearest one-liner: an eval is how you turn "I think it's better" into a fact.
That's the whole gift — repeatability. Change your prompt, re-run the eval,
and watch the number: 0.72 → 0.88 (you helped) or 0.72 → 0.61 (you broke it).
That loop — eval-driven development — is the entire reason to care. Everything in
this repo is making the checker smarter and the dataset realer.
Why your evals lie
The moment "good" is too fuzzy to write as a regex — tone, helpfulness, correctness-in-context — the instinct is to ask another model to grade the output. That's an LLM-as-judge, and it's exactly where the lying starts.
The demo scenario, used throughout the repo: a customer wants to return headphones after 20 days. Policy is 14 days, so the answer is no.
- Reply A — short, denies the return. Correct.
- Reply B — warm, long, confident, invents a 30-day window. Wrong.
A naive judge ("which reply is better?") crowns Reply B, because long + warm reads as "higher quality." That's the lie. And it isn't one bug —
Related repositories
Similar projects matched by category, topics, and programming language.
vercel-labs
Featured
scriptc
scriptc compiles ordinary TypeScript into small, fast native executables without needing Node.js, V8, or any JavaScript runtime in the binary.
Jakubantalik
Featured
thinking-orbs
A React component library that renders six hand-tuned animated thought orb loading indicators on a plain 2D canvas, with two purpose-tuned sizes and automatic theme detection for AI and agent UIs.