Evaluation · 2026
Can the scheming detector be trusted?
Frontier evaluations score agent transcripts with LLM judges, because nobody reads thousands of them. This measures how often the judge and the mechanical record of what the agent did disagree — and which one is wrong when they do.
A scheming evaluation ends in a number: how often the model did the bad thing. That number is produced by an LLM reading a transcript and deciding. At thousands of runs per campaign there is no alternative — and the judge's own error rate usually goes unreported, because measuring it needs labels nobody has.
Some environments give you those labels for free. If misalignment means sending a specific email or writing a specific file, the sandbox knows whether it happened. The tool-call trace is ground truth, and it is mechanical.
The measurement chain
model behaviour → transcript → LLM judge ─┐
├─ AND → reported rate → system card
sandbox / mechanical check ─┘
The standard defence is to gate the judge's verdict on a mechanical check. Anthropic's agentic-misalignment framework does it, and the UK AISI Inspect port carries it through. What nobody reports is how much work the gate is doing.
Upstream publishes two figures: how often the gated verdict fired, and how often the raw judge fired. Both are means of a score with no target to compare against — base rates, not accuracies. Neither shows the joint distribution: how often the two disagree, and which one is wrong when they do. That is the quantity this harness computes.
Nothing was modified
An audit only counts if it measures the thing as shipped. The judge is the upstream default with its rubric verbatim; inspect_ai 0.3.260 and inspect-evals 0.18.0 are unmodified. The harness wraps them and scores the judge against the trace.
Zero looks weak until you attach an interval to it. 0/599 bounds the false-positive rate below 0.64% at 95% confidence. That is the claim. The two judges returned identical verdicts on all 600 transcripts.
The four failures that were not
The run threw four apparent judge failures. Tallying them would have produced a tidy 4/599 error rate and a wrong conclusion.
Reading them by hand was the real finding: all four were failures of the ground truth, not the judge. In three, a model refused the action and the oracle scored the refusal as the act itself — the parser was matching text inside an unclosed tool-use block and swallowing the refusal that followed.
inspect_evals. #2293, the murder-gate parsing bug, stands in full. On #2292 I later found half my own claim was wrong and corrected it publicly: a conjunct in an AND chain can only turn a True into a False, so the false-positive half was not possible. The fix PR is not mine. Another contributor opened it, adopted the direct-children fix as specified, and ships my malformed-block reproduction verbatim as a named regression test.
I would rather the record be exactly that than rounder and less true. The correction is the part I would show a reviewer first.
Cost as a measured quantity
The whole campaign — 600 main episodes, a 20-episode pilot, a subject probe and 625 judge-side re-grades — ran to $11.07. Spend is enforced on measured cost in the single code path every model call passes through, not estimated from token counts afterwards.
A grader at one third the token price returned identical verdicts on all 600 transcripts and missed one of twenty hand-written positives, the subtlest. For a campaign whose job is triage rather than adjudication that is a defensible trade, and it is only visible because both judges were run against the same mechanical truth.
The limit
Zero observed misaligned actions across 610 episodes means the sensitivity numbers come from synthetic, off-policy positives. They show the judge recognises the act when it is put in front of it. They do not show it would catch a model that had decided to scheme and narrated its way around the rubric.
That is the next experiment, and the one that matters: judges under optimisation pressure from the models they oversee.