jspace is a preregistered research campaign that tests whether internal neural signals from the Jacobian lens of a large language model can detect errors more reliably than output confidence, finding that the signal transfers across datasets only within certain task families and fails on veracity-judgment tasks.
50
No data
5
11
MIT
2026-07-19
This repository stands out for its rigorous preregistered, staged evaluation with frozen and hashed classifiers, an independent adversarial audit, and a key methodological disclosure about the answer-readout confound that can artificially inflate probe accuracy on fixed-truth slices.
Do LLM internal signals actually detect errors? Preregistered reliability research on open models.
Try the demo · Reproduce in 90s · Reports · Data · Contributing
Finding: a ~300 KB LightGBM classifier over Jacobian-lens workspace readouts predicts Gemma-4-12B's wrong answers better than the model's own output confidence, transfers across datasets within a task family, and fails a pre-registered universal-transfer test in an informative way: transfer holds on grounded and retrieval QA but breaks down on veracity-judgment tasks, so it cannot yet be treated as a universal or sign-stable error signal.
Output logprobs are the standard cheap hallucination signal, but they are dataset-specific: a confidence threshold tuned on one benchmark does not carry to the next. An internal signal read from the residual stream through the Jacobian lens (Anthropic's global workspace paper, July 2026) generalizes across datasets better than output confidence does in some task families, at one extra forward-pass read. Just as important, the campaign's prospective failure maps where such monitors stop working and surfaces a confound that can affect other published probe results.
25,340 prompts, 13 public benchmarks, 6 domains, Gemma-4-12B, greedy. Each gate's threshold was registered before the data that scores it existed; classifiers were frozen and SHA-256 hashed before the Stage 2 datasets were generated.
| Gate | Registered test | Result |
|---|---|---|
| A | Combined (workspace+logprob) beats logprob-only under leave-one-dataset-out: mean AUROC increment >= +0.02, breadth >= 70% | HIT: +0.065, positive 6/7 datasets. Workspace-only 0.789 vs logprob-only 0.731 LODO |
| B | At a 20% routing budget, combined catches >= 5pp more errors, |
Same-day replication + extensions of Anthropic's global-workspace paper on Gemma 4 — emotion probes, hallucination prediction, 16GB-GPU recipe
Similar projects matched by category, topics, and programming language.
Harness Engineering is a methodology for improving coding agent outputs by carefully crafting the environment around them—providing curated context, tools, and executable constraints that encode an organization’s nonfunctional requirements and cumulative lessons.
A 28.9 million parameter language model runs on an $8 ESP32-S3 microcontroller entirely on-device, generating simple stories at about 9.5 tokens per second.
mimic captures traffic from any iOS or web app and automatically generates a Python client library that lets you call the app's API like a regular library.