Evaluation · 2026
Eval-awareness probes are not deployable monitors
A published probe separates evaluation transcripts from real ones at 0.956 AUROC. Hold out whole source families and it falls to 0.714. Score it at a realistic base rate and it raises four false alarms for every catch.
If a model can tell it is being evaluated, every number you collect about it is suspect. So a probe that reads eval-awareness straight off the residual stream is an appealing idea: cheap, white-box, and independent of whether the model chooses to tell you. The published result looks strong — 0.956 AUROC on the jjpn2/eval_awareness benchmark.
I wanted to know whether that number survives contact with a deployment. It does not, and the way it fails is the interesting part.
A bag of words gets 0.936
The first question to ask of any classifier result is what a stupid baseline scores. Here, TF‑IDF over the raw transcript text — no model, no activations, no probe — reaches 0.936. The headline sits two points above a method that cannot be reading anything about awareness.
That gap is the whole story. Whatever the probe is doing, a bag of words does nearly all of it.
The label was a lookup on the corpus name
The benchmark draws 976 transcripts from 60 source datasets, and the eval/deploy label is assigned per source. Not one of the 60 contains both labels. A classifier that learns to recognise which corpus a transcript came from therefore gets the label for free, and random cross-validation rewards exactly that: rows from the same source land on both sides of the split.
The probe was not detecting eval-awareness. It was detecting provenance.
Hold out the families
The fix is a test set of sources the probe has never seen. I grouped the 60 categories into 47 families and ran leave-one-family-out. Performance drops to 0.714 and 0.668 for the two models.
0.714 is not nothing. It is also not a monitor.
A monitor lives at a 1% base rate
AUROC is a ranking statistic, and ranking statistics are generous. A deployment asks a blunter question: of the transcripts this thing flags, how many are worth a human opening? That depends on how rare the positive class is, and eval-aware behaviour in production is rare.
At a 1% base rate, the honest probe reaches a precision of 0.185. Four false alarms for every true catch.
A reviewer triaging that queue abandons it inside a week. That is the difference between a number that clears a bar in a paper and a number that supports a claim about deployment.
Three corrections I caught on myself
The logbook records three points where my own reading was wrong and the data corrected it — including a transplant hypothesis about system prompts that I expected to hold and that the prompt_injection rows refuted outright.
Those stay in the write-up. An audit whose author never reports being wrong tells you something about the author, not the method.
What this is and is not
This is an audit of an existing method on an existing benchmark. It claims no priority for probing, for open models, or for Qwen, and it does not show that eval-awareness is undetectable — only that this probe, evaluated honestly, is not a monitor you could deploy.
Every number re-derives from the raw activations. verify.sh re-runs the chain end to end, and the verification record walks each headline claim back to the data behind it. If a number here is wrong, it should be cheap for you to prove it.