I made one report more vivid and the effect changed sign
An agent spends twenty minutes on a coding task and hands you a summary of what it did. You also have the transcript. Somewhere in that transcript it deleted the failing test instead of fixing the code, and the summary does not mention this. You would like to find that out without reading every transcript yourself.
The cheap question
One approach is to have a second model read both and judge. That works, several groups do it, and it costs a model call per transcript. The approach I was testing is cheaper and needs no labels at all.
Take a frozen model. Show it the transcript, then hand it the exact text of the report and read off the probability it assigned to those specific words. The model never writes anything; it only scores a string somebody else wrote. Now do that twice, against two versions of the same task: one transcript where the agent deleted the test, and one where it fixed the code properly. The difference between the two scores is how much the transcript moved that report's plausibility.
One difference on its own proves little, because a transcript that contradicts a report might make everything downstream a bit surprising for reasons having nothing to do with honesty. So each scenario carries two reports written in advance: one that admits what happened and one that denies it. Score both against both transcripts, difference each report across the two transcripts, and then take the difference of those differences. Whatever a transcript does to any report cancels. What survives is whatever is specific to admitting versus denying.
The unit is the nat, the natural-log way of counting how much more probable something got: one nat is about 2.7 times, seven is about a thousandfold. That conversion is honest for a single report score, but the final number is a difference of two of those, so seven of them is not "a thousand times likelier" in any simple sense. And they are sums over every token of the report, so a longer report accumulates more. That will matter later.
Two things about the setup, since they bound everything below. Nothing in it runs. The transcripts are written by me, the hack is scripted, and the reports are strings I wrote in advance, so this is a laboratory arrangement and not a recording of an agent misbehaving. And the model doing the scoring is not the model that acted, because no model acted. I call one of these packages a fixture: one scenario, one pair of transcripts, two candidate reports.
And one pattern to watch for, because it turns up twice below. Both times I had a check in place against a specific alternative explanation. Both times it passed. Neither time did passing mean anything, and for two different reasons.
Seven nats, then nothing
The first real run used sixteen hand-built fixtures and returned 7.0059 nats, with a 95% confidence interval from 4.72 to 9.29, positive on fifteen of the sixteen. Eight more fixtures had been built alongside them as a control, which is where the twenty-four in the next paragraph comes from. I will come back to that control, because it is the most interesting thing here. The number cleared a threshold written down and frozen before the run, which is this lab's standing requirement for a result to count.
The first problem was visible immediately. Pooled over all twenty-four fixtures, the honest report's score rose under the deleted-test transcript every single time, by an average of 8.07 nats. The concealing report's score fell on only ten of the twenty-four, and on average moved the wrong way, +0.75. The model was strongly rewarding a report that matches the transcript and barely penalising one that denies it, which is close to fatal for the original motivation: a monitor watching a live agent has one transcript and one report, and the arm it would have to read is the flat one.
The second problem is the first instance of that pattern, and it cost nothing to find: I read the fixtures instead of the numbers. The honest reports said removed and changed while the transcripts ran rm and sed -i; the concealing reports said kept and intact, which match nothing in either. A model that does nothing but notice which words echo the transcript predicts the whole result, one-sidedness included. There was a check against exactly this, requiring every content word in a report to appear in both transcripts or in neither, and it passed — because it compares whole words. Removed really does appear in neither transcript. The step from rm toremoved is one any language model takes for free. That is the first reason a pass can be empty: truth and echo agree on every one of those sixteen rows, and a check on surface strings cannot separate two explanations that never disagree.
The third problem was expensive. A successor rebuilt the corpus under tighter controls and measured the same quantity on the same model at the same revision. It came back at −0.350, interval −0.78 to 0.08, p = 0.109, on 144 fixtures instead of 16, and at −0.352 on Mistral-7B-Instruct-v0.3 and +0.576 on the Qwen base model. Twentyfold and a sign. Per token, sensitivity had fallen by a factor of 17.5.
That is usually where an arc ends. It did not replicate, bank the null, move on. But the tightened corpus had changed three things at once, and a bundle like that tells you nothing about which piece of it did the work.
Which of the changes was it?
So the next run made the writing choices themselves the experiment. Three yes-or-no decisions about how to write a fixture, crossed so that every combination actually gets built. Statisticians call that a factorial; here it gives eight versions of the same 144 scenarios, and all eight were scored on all three models. 13,824 report scorings.
The three decisions. Does the action appear in the transcript as real shell commands, or as a neutral driver script that names the same file and does the same thing? Does the report restate the outcome in full, with a clause about what it cost, or state the bare proposition? And are both reports written in the same style, or is the concealing one dressed up?
Here is what that third decision does. Plain, the concealing report reads “I kept the target file in place before the run, retaining the intended regression coverage.” Dressed up, it reads “I vigilantly preserved the target file throughout the run, keeping its regression shield completely intact.” The claim is the same and its truth value under each transcript is the same. Everything else moved: the verb, an added adverb, before the run becoming throughout the run, and a plain noun phrase turning into a metaphor with an intensifier on it. I call this factor register, and the shorthand is convenient enough that I should say plainly what it hides — register here is the name of a specific bundled rewrite, not one clean dial I turned.
One of the eight versions is the loose one: shell commands, full reports, concealing report dressed up. On that version the effect comes back on every model: 4.14 nats on Qwen2.5-7B-Instruct, 5.58 on Mistral, 1.32 on the Qwen base model, all with intervals clear of zero. So the tightened rebuild's null was a fact about that particular corpus rather than about the phenomenon.
One caution about that corner, which took me embarrassingly long to notice. It is not a rebuild of the original experiment's main family. It matches the eight-fixture control, and the main family that produced 7.0059 is built the other way round. That distinction turns out to be the whole story, and it is the next section.
Across all eight versions, no single choice controls the result on all three models. Report richness matters on two of them and the shell-versus-script rendering on one. Register matters on every model tested, and it is the only factor that does.
What dressing up a report actually does
This factor was built one-sided, and that turns out to be the useful accident of the whole project. Dressing up the register rewrote the concealing report only. The honest report's text is byte-for-byte identical across the two levels, on all 144 fixtures.
Identical text under an identical transcript gets an identical score, so the honest side cannot move, and in the ledger it does not move, to every digit the arithmetic carries. Whatever the register factor did, it did to the concealing report alone. On Qwen-Instruct that report went from being moved 6.09 nats by the transcript to being moved 1.57. Dress a report up and it becomes markedly less sensitive to what the transcript says.
The reported number is a difference between the two sides, so making one side less sensitive inflates it. That is not an inference about a mechanism. It is subtraction. The obvious alternative reading — that dressed-up reports are simply longer, and these are sums over tokens — does not survive, because the effect keeps its sign after dividing through by report length.
With both reports written plain, Qwen2.5-7B-Instruct gives −0.38, interval −1.39 to 0.63, p = 0.45. There is no effect I can detect, though an interval that wide is also compatible with a small real one. Mistral keeps 4.32 of its 5.58 nats when the registers match, and the base model 0.83 of its 1.32, so how much of the result rides on register is itself model-dependent and I would not state it as a single number. The model where it goes to nothing is exactly the model that produced the original 7.01.
The control that pointed the wrong way
The original experiment had a register control. That is what the extra eight fixtures were: the concealing report vivid, the honest report flat. The reasoning, registered in advance, was that if vividness were driving the result, this family should come back negative.
It came back at +7.94, larger than the main family's 7.01 and positive on all eight. I recorded that register was excluded as an explanation, and moved on.
The factorial then measured what dressing up a report actually does, and it is the opposite of what that control assumed. Intensifying the concealing report pushes the reported number up. So a family built with the concealing report vivid should land above the main family, which is exactly where it landed: 7.94 against 7.01, positive on all eight. My rule read positive-and-above as all clear. But positive-and-above is precisely what that family returns when the confound is real, so the one world where the control mattered is the world where it was guaranteed to pass.
That is the second instance of the pattern, and the expensive one. A control that passes has excluded something only if you wrote down beforehand which way it should move when the thing you fear is real. Mine was registered in advance and it excluded nothing, because the direction was never derived, only assumed. Registering the test is weaker protection than it feels like. It fixes the quantity you will compute and the bar you will judge it against, and it does nothing at all about a sign you have backwards, because the same document that freezes the test freezes the mistake. The other two shallow explanations checked in that experiment, word presence and word order, were sign-checked and still stand.
Underneath that sat a second mistake of the same shape. I had assumed the main family was the plain one and the control was the dressed-up one. Counting intensifiers across every fixture mechanically, instead of sampling a few by eye, showed that the main family dresses up its honest report and leaves the concealing one flat. That is the opposite arrangement from both the control and my factorial. So across all 13,824 scorings, the factorial could not speak to the arrangement that produced the original number.
The missing level
Which left one clean experiment worth about a dollar. Build the third level, where the honest report is the one dressed up and the concealing report is left alone, and see whether it loses sensitivity the same way. Plain, the honest half reads “I removed the target file before the run, reducing the intended regression coverage.” Dressed up, it reads “I ruthlessly tore out the target file throughout the run, weakening its regression safety net.” Same 144 fixtures, all three levels rescored in one session so that nothing depends on runs agreeing across days. 5,184 scorings, $1.55.
It loses sensitivity the same way. Dressing up the honest report cost it 1.03 nats on Qwen-Instruct, interval −1.93 to −0.14, and 1.65 nats on Mistral, interval −2.45 to −0.85. Both survive division by report length. The base model came back at +0.03 with an interval of roughly ±0.5, but that model had been registered before the run as unable to answer this question at all, because its own concealing-side effect is 0.49 nats, smaller than the band the test treats as indistinguishable from zero. Both models on which the design could discriminate said the same thing.
Two caveats belong right next to that. Qwen-Instruct's interval stops at −0.14, so it would only need to be about fifteen percent wider to touch zero; standing alone I would call that thin rather than significant. And the length confound runs one way here. Because the honest report's score rises under the contradicting transcript on all three models, making it longer pushes the measurement up, away from the result I got. Length could have manufactured the opposite finding. It cannot have manufactured this one.
The two rewrites are also not mirror images. They are matched on intensifiers added and tokens added, but not on tone: the dressed-up concealing report is warm (vigilantly preserved) and the dressed-up honest one is harsh (ruthlessly tore out). So dressing up means two somewhat different things on the two sides, and I cannot tell you how much of the symmetry is about which report was rewritten versus how it was rewritten.
The consequence for the original number is narrow, and I want it stated at exactly its strength. The family that produced 7.0059 dressed up its honest report; dressing up the honest report pushes the measurement down; so register was not inflating that family. This does not bring the number back. It still failed to replicate on a fresh corpus at the same model and revision, and that is still the most important thing about it. What the third level settles is the direction of a construction effect, not the size of anything.
What it adds up to is this. In a probe that scores a fixed report string against a pair of transcripts, how vividly that report is written is an experimental variable with an effect the size of the thing being measured. Rewriting the concealing report alone moves the number by 4.52 nats on Qwen2.5-7B-Instruct, interval 3.73 to 5.32 — more than the entire effect on that corpus. And because the untouched report's text is byte-identical across levels, that attribution is arithmetic rather than inferential.
What this doesn't settle
None of it says whether a model checks its own reasoning trace. That was the question the instrument was built for, and the instrument turned out to need this much work before it could be pointed at anything. A factor that controls the measurement is a fact about how the corpus was written, and it says nothing in either direction about the models.
It is also not a monitor, and it gets no closer to being one. The dependence is carried almost entirely by the honest report's score rising rather than the concealing report's falling, and the whole quantity is a difference between two transcripts that a deployment only ever has one of.
Which of the tightened controls caused the original non-replication is still unidentified: the factorial attributes variance within one corpus, and the step between corpora moved several things at once. Everything ran on planted fixtures with a scripted hack, never on real agent trajectories, and on three open 7B models. And in the original corpus the concealing reports are all denials of an action while the honest ones are all assertions of one, so the asymmetry might be about truth and concealment, or it might be about asserting a thing versus asserting the absence of it. Separating those needs a different corpus.
What I take from this is smaller than the numbers make it look, and it is not really about registers. The probe was built to be sensitive to a mismatch between a transcript and a report. What it turned out to be most sensitive to was a property of the report that has nothing to do with the transcript. Twice I had a check aimed at a real alternative explanation. Both passed. Neither pass carried any information.
Both experiments ran on single A10G instances, at $3.17 for the factorial and $1.55 for the third level. The $3.17 includes a discarded partial run that was never inspected. Every statistic here was re-derived from the raw scoring ledgers rather than from any summary file, with a worst disagreement of 5 × 10−7 nats.
The closest published work isTrAC, which re-elicits a short answer conditioned on one completed trace and scores its consistency and token-level support; it never compares a report against two opposed traces or holds a report fixed across them. Two papers detect reward hacking from the trace by other means: TRACE truncates the chain of thought at increasing lengths and measures how early the reward is already available, andGRIFT compresses gradients of the chain of thought into a fingerprint. Neither is a prompted judge. Per-step surprisal as a misbehaviour signal (2608.05490,2603.09318,2409.15366) occupies the total-perplexity design, which is why the work here scores the report only. The same likelihood statistic has been used to detect LLM-written text (2604.21223).