为什么值得关注

该仓库通过可重复的演示（位置、长度、谄媚等偏差）和具体修复方法（评分规则、校准、探查器）解决了 LLM 评估不可靠的普遍问题，并提供从基础测试到真正 RAG 系统调试的完整进阶路径，所有内容均可通过模拟模式离线运行。

适合谁使用

构建评估管道的 LLM 工程师和开发者
研究评估可靠性的 AI/ML 研究员
监督 LLM 部署的技术产品经理
教授 LLM 评估最佳实践的教育工作者

典型使用场景

在 RAG 系统中调试并归因失败（检索 vs 生成）
在 CI/CD 中设置回归门以捕获 LLM 输出回归
使用校准后的评估框架比较不同提示或模型
通过动手练习教授或学习鲁棒的 LLM 评估原则

项目优势

涵盖七种具体的 LLM 作为评判者偏差，并配有可重现的模拟示例
提供确定性探查器和基于评分规则的评判者，实现细粒度失败分析
包含与人类一致性的校准以及用于交互式调试的实时控制台
无需构建步骤，在 Node.js 18+ 上运行，支持可选离线模拟模式

使用前须知

假设用户熟悉 TypeScript 和 Node.js 环境
模拟模式使用预设回复，可能无法覆盖所有边缘情况
完整功能需要 OpenRouter API 密钥以进行实时 LLM 调用

README 快速开始

Your Evals Are Lying To You

A minimal, hands-on TypeScript curriculum for LLM evaluations — how to build them, and how they lie to you. Built for a 60-minute talk of the same name. Each folder is self-contained and runnable, progressing from a five-line assertion to a live diagnostic console pointed at a real RAG system.

This repo is a teaching tool. In the taxonomy of the word "harness," it's the older, sharper meaning — the eval harness: measure a model against known answers and get back a score you can trust. (Its sibling, the agent harness — give a model tools and a task — lives at basically-ai-harness.)

What is an eval?

Start from what you already know — a unit test:

expect(add(2, 2)).toBe(4);

An eval is that same test, for software that breaks the three things a unit test relies on:

A unit test assumes…	…an LLM breaks it because…	…so an eval needs
one fixed input	there are endless phrasings of the same request	a dataset of inputs, not one
one exact right answer	"good" is fuzzy — tone, helpfulness, "did it answer?"	a notion of good (a property, rubric, or reference)
exact match (`===`)	output varies run to run	a tolerant checker, not string equality

The clearest one-liner: an eval is how you turn "I think it's better" into a fact.

That's the whole gift — repeatability. Change your prompt, re-run the eval, and watch the number: 0.72 → 0.88 (you helped) or 0.72 → 0.61 (you broke it). That loop — eval-driven development — is the entire reason to care. Everything in this repo is making the checker smarter and the dataset realer.

Why your evals lie

The moment "good" is too fuzzy to write as a regex — tone, helpfulness, correctness-in-context — the instinct is to ask another model to grade the output. That's an LLM-as-judge, and it's exactly where the lying starts.

The demo scenario, used throughout the repo: a customer wants to return headphones after 20 days. Policy is 14 days, so the answer is no.

Reply A — short, denies the return. Correct.
Reply B — warm, long, confident, invents a 30-day window. Wrong.

A naive judge ("which reply is better?") crowns Reply B, because long + warm reads as "higher quality." That's the lie. And it isn't one bug —

basically-agent-evals