该仓库因其严格的预注册分阶段评估（含冻结和哈希验证的分类器）、独立对抗审计，以及揭示“答案读出混杂”这一关键方法论问题而引人注目。
Do LLM internal signals actually detect errors? Preregistered reliability research on open models.
Try the demo · Reproduce in 90s · Reports · Data · Contributing
Finding: a ~300 KB LightGBM classifier over Jacobian-lens workspace readouts predicts Gemma-4-12B's wrong answers better than the model's own output confidence, transfers across datasets within a task family, and fails a pre-registered universal-transfer test in an informative way: transfer holds on grounded and retrieval QA but breaks down on veracity-judgment tasks, so it cannot yet be treated as a universal or sign-stable error signal.
Output logprobs are the standard cheap hallucination signal, but they are dataset-specific: a confidence threshold tuned on one benchmark does not carry to the next. An internal signal read from the residual stream through the Jacobian lens (Anthropic's global workspace paper, July 2026) generalizes across datasets better than output confidence does in some task families, at one extra forward-pass read. Just as important, the campaign's prospective failure maps where such monitors stop working and surfaces a confound that can affect other published probe results.
25,340 prompts, 13 public benchmarks, 6 domains, Gemma-4-12B, greedy. Each gate's threshold was registered before the data that scores it existed; classifiers were frozen and SHA-256 hashed before the Stage 2 datasets were generated.
| Gate | Registered test | Result |
|---|---|---|
| A | Combined (workspace+logprob) beats logprob-only under leave-one-dataset-out: mean AUROC increment >= +0.02, breadth >= 70% | HIT: +0.065, positive 6/7 datasets. Workspace-only 0.789 vs logprob-only 0.731 LODO |
| B | At a 20% routing budget, combined catches >= 5pp more errors, |
Same-day replication + extensions of Anthropic's global-workspace paper on Gemma 4 — emotion probes, hallucination prediction, 16GB-GPU recipe
根据分类、Topic 和编程语言匹配的相似项目。
Harness Engineering is a methodology for improving coding agent outputs by carefully crafting the environment around them—providing curated context, tools, and executable constraints that encode an organization’s nonfunctional requirements and cumulative lessons.
A 28.9 million parameter language model runs on an $8 ESP32-S3 microcontroller entirely on-device, generating simple stories at about 9.5 tokens per second.
mimic captures traffic from any iOS or web app and automatically generates a Python client library that lets you call the app's API like a regular library.
