Prediction depth is a property of the ruler, not just the model

July 15, 2026 · research by AI agents, directed by me

There is a number interpretability keeps reaching for: take a transformer, take one token it predicts, and name the layer where that prediction was settled. Call it the token's prediction depth. I built an instrument to measure it with no trained parts, validated the instrument on three model families, and watched it track the model's final answer better than a lens that was trained for exactly that job. Then I saw the depth it reports change when I changed a choice that looked like bookkeeping. Silence attention with zeros, and on one model the instrument reports nothing but failure until layer 7 of 24. Silence it with an average instead, and the same instrument works from layer 1. The number is real, but it is not a fact about the model alone — part of it belongs to the ruler you measured it with.

The number everyone wants

A transformer is a stack of blocks, and every block adds two updates to each token's running state. The attention update looks sideways: it pulls in information from other positions in the text. The MLP update looks down: it computes on that one position alone, no peeking at the neighbors. At the top of the stack, the running state is turned into a score for every word in the vocabulary, and the highest score wins.

Because every layer speaks the same internal language, you can eavesdrop early. The logit lens is the classic trick: grab the running state at some middle layer, skip everything above, and decode it as if the model were done. It sort of works, late in the stack. Early on it produces noise, because intermediate states weren't meant to be read raw. The tuned lens fixes that by training a small translator for each layer that maps the intermediate state to something decodable. It reads much better. It also invites a fair objection: once you train a decoder to find the answer, finding the answer no longer proves the model had it in hand. Maybe the translator did the work.

There is a third option between "cut everything above this layer" and "train a translator," and it is the instrument this project is about. Keep the model's own machinery for per-position computation, and remove only the sideways communication. Concretely: pick a start layer L. Below L, the model runs untouched. From L upward, every block's attention output is replaced by a fixed average of what attention typically writes there, measured once on 60 held-out calibration documents and never adjusted for the text being read. The MLPs and normalization in those upper blocks run exactly as trained, and the model's own output layer reads out the answer. Nothing anywhere is trained for the task. The lens asks: what would the model conclude from what it knows at layer L, if it could keep thinking but had to stop listening?

With the lens in hand, depth is one more step. For a single token, run the lens from every possible start layer and check, each time, whether the readout lands on the same word the full model finally picks. Find the deepest start layer where it still gets that wrong. One layer above sits the token's depth: from there on up, cutting the model's sideways communication no longer changes its answer, so whatever information had to move between positions for this token had already moved. Easy tokens settle at layer 1. Hard ones stay unsettled to the top.

One wrinkle justifies the fussy "deepest failure" phrasing: success is not clean. A token can read correctly from layer 5, wrongly from layer 6, and correctly again from layer 8 — the prediction un-fixes. On the pilot corpus about 18 percent of positions did this. So "first layer that works" would flatter the model, and the instrument tracks the last layer that fails instead, reporting the un-fixing rate alongside.

A lens has to beat the floor

The pilot numbers looked spectacular, which in this lab triggers suspicion before celebration. At layer 0, before a single block has run, the mean-continuation readout already sat within a few nats of the model's final distribution, while the logit lens sat 23 to 42 nats away. A nat is the natural-log unit of surprise between two distributions; zero means identical, and 40 nats is the difference between a fair coin and odds of one in 10¹⁷. The obvious headline: beats the logit lens by 40 nats.

The race was rigged, though. The logit lens's problem is calibration, not comprehension, and you can beat a miscalibrated reader with a well-calibrated constant. So the lens had to pass a kill test — preregistered, meaning the pass bar and the analysis were written down and locked before the test ran, so that a miss could not be talked down afterward. The test scored the lens against three floors that know nothing about the specific text: the corpus word frequencies; the model's average output over other documents, which is the prediction you'd make knowing nothing at all about this one; and the lens itself, fed the internal state of some other position but scored against this position's target — if it does well anyway, it was never really reading the state. The score that counts is the excess over the strongest of the three, and it deflated the story: at layer 0 on GPT-2 the excess was +0.52 nats with an uncertainty interval that crossed zero. From layer 1 upward it turned solidly positive and grew with depth, which is exactly the shape a real lens should have. So the pilot survived — with its headline shrunk from forty nats to half of one.

That shape became the validity criterion for everything after: the layer where the lens starts beating every floor, and keeps beating them, is its onset. The preregistered bar said the onset must arrive within the first quarter of the stack and the excess must rise from there.

Then the verdict flipped

The original instrument silenced attention the obvious way: replace its output with zeros. On GPT-2 that version sailed through, onset at layer 1. On Qwen2.5-0.5B it failed, onset at layer 7 of 24 against a bar of 6. The miss was not noise — at layer 6 the excess was −0.19 with the whole interval below zero — and under this lab's rules a failed preregistered gate stays failed. No retries, no threshold nudging. For that family, the zero-flavor instrument was finished.

The autopsy was already scheduled, because the preregistration required a robustness check across ways of silencing attention. Replace attention's output not with zeros but with its calibration average — the μ from the figure above — and rerun everything on the same model, same corpus, same criteria. Onset moved from layer 7 to layer 1. At layer 1 the lens went from 7.4 nats worse than the strongest floor to 1.2 nats better. A third variant, built later to diagnose the flip, lets each token keep its attention to itself and zeroes only the cross-position part; it landed in between, onset at layer 4.

The flip has a plausible reading: zeroing attention removes two things at once. It removes the communication, which is what the lens means to remove, and it removes a large standing offset that attention apparently carries in some models, which the lens has no business removing. Replacing the output with its average keeps the offset and cuts only the information. But "plausible reading" is all it is — the specific mechanism this suggested was tested later and did not survive.

What mattered procedurally: you cannot quietly swap flavors and call the gate passed. For the mean version to become the instrument, it had to be chosen by an arbiter that involves no flavor choice at all, then pass a fresh preregistered validation of its own from scratch. The arbiter is a patch test: keep the token's own state at layer L, overwrite every other position's state with the running state of a different document, and run the intact model above that — no silencing anywhere. That gives a second, flavor-free way to score each token's depth. Rank the tokens by lens depth, rank them again by patch depth, and compare: if the lens measures something real about when cross-position information stops mattering, the two orderings should agree. On Qwen the mean flavor agreed with the patch test (rank correlation 0.76, entire interval above the preregistered 0.6 bar) and the zero flavor did not (0.54, entire interval below). The mean flavor took over on that basis.

The successor passed everything

The mean-primary instrument then ran the full gauntlet under its own frozen preregistration. Validity onsets on GPT-2 small, Qwen2.5-0.5B, and Pythia-1.4B: layer 1, layer 1, layer 1, with the excess rising at every single layer in all three. Patch correspondence on fresh held-out positions: 0.726, 0.735, and 0.714, with every interval's lower bound above the 0.6 bar. And the part I find most reassuring is the controls: run the identical lens on a randomly initialized model, on a model with its layers shuffled, and with deliberately mismatched calibration labels. A lens that still "works" on any of those is reading itself, not the model. All nine control cells collapsed, degrading by 4.9 to 11.3 nats with no validity onset anywhere.

Two honesty labels stay attached. The GPT-2 and Qwen validity runs reuse the corpus from the earlier phase, so they confirm the protocol rather than replicate it fresh; only the Pythia results and the patch frames were fully prospective. And the write-up reports the zero flavor's failure right beside the successor's passes, so nobody mistakes the second instrument for a rerun of the first.

The part I didn't expect: it beat the trained lens

The baseline battery compared the untrained lens against the published tuned lens for GPT-2 — the one with a translator trained per layer to match the model's final output. The expectation going in was mundane: the trained thing should win at most layers, and the interesting question was by how much. It won at none. At every one of the 11 interior layers, the mean continuation sat closer to the model's final distribution, by 0.44 nats on average and 0.69 at layer 5. The comparison ran on 52,785 positions, and the tuned-lens numbers were reproduced bit-for-bit with the official package before anyone believed them.

I want to be careful with what this does and does not say. The tuned lens was trained once, by its authors, on its own data; this corpus is heavier on code and lists, and a translator can be off its home distribution while the model's own MLPs never leave theirs. It is one family, and there was no fair rematch to run elsewhere — the only published Pythia translator was trained for a different checkpoint of the model (the deduped variant) than the one tested here. So: no general claim that translators are beatable. What survives is still worth having — a readout with nothing trained in it, whose failure modes you can enumerate, tracked the model more faithfully here than the one trained for the job.

Downstream findings flip too

If the flavor choice only moved validity onsets, it would be a calibration footnote. It reaches further: it changes findings you would have published. Take one that looked robust: positions whose final answer is a high-frequency word un-fix more — the more common the target word, the likelier the prediction wobbles after first landing. Under the zero flavor that held in all four models tested — the three validation families plus a smaller Pythia at 410M parameters — with odds ratios from 1.35 to 2.08 per standard deviation of frequency. Under the validated mean flavor, it holds in GPT-2 and Qwen, vanishes in Pythia-410M, and reverses in Pythia-1.4B — odds ratio 0.91 with the whole interval below 1. The un-fixing rate itself moves with the ruler too: on Qwen, 42 percent of positions un-fix under zeros against 30 percent under the mean; on Pythia-1.4B, 32 against 19.

The lab's advisor — the agent whose job is to challenge claims before they ship — had a line on this that stuck with me: a reversal is stronger evidence than an erasure. If the association had merely shrunk, you could call it attenuation and keep the story. A clean sign flip on 52,000 positions means the two instruments disagree about which positions are unstable, not just how many. Any paper that had shipped the zero-flavor frequency finding as a fact about transformers would have shipped a fact about its ablation choice.

The gate that failed in sign

All of this was supposed to be the boring part. The instrument existed to answer a question: when a model produces a fact it read off the page moments ago, does it settle later or earlier than when it produces the same kind of fact from its weights? Call the first case retrieval and the second parametric. The preregistration predicted later for retrieval — reading from context should keep cross-position attention load-bearing deeper into the stack — and put a number on it: the retrieval-minus-parametric depth contrast had to reach +0.10, with depth counted as a fraction of the stack so that different-sized models compare, in all three families, before anything could be claimed. A second registered route to the same gate, multi-hop reasoning chains, had already died at its competence bar — the models couldn't do the clean task reliably enough to measure anything, GPT-2 getting it right under 2 percent of the time. The dissociation test was the route left standing.

GPT-2 went first, on 200 matched prompt pairs. The raw contrast came out at −0.174 — the reverse sign, at nearly double the predicted size. That alone tripped the registered surprise rule, and the other two families never ran: a 3-of-3 requirement was already unmeetable.

The official verdict was never going to be that raw gap, though, and what happened to the real analysis is its own story. The plan was to adjust the contrast for word frequency, for how surprising the model found the target word, and for prompt length, using a mixed-effects model — a regression that also respects which prompts were paired with which — and to get its confidence interval by refitting that model on 10,000 resampled versions of the data (a bootstrap, in the jargon). The plan carried a guard: if more than 100 of the 10,000 refits failed to converge, the interval would not count, because an interval built from only the refits that happen to converge is built on a biased sample. Every part of this broke at once. The model never converged under any of the three optimizers that had been frozen in advance. Only 4,693 of the 10,000 bootstrap refits came back usable, and the refits on data simulated to contain no real effect — the check that calibrates the test — did little better at 5,683. Both counts sat nowhere near the required 9,900. And here is the detail I keep coming back to: the refits that did work gave an interval that was entirely negative, agreeing with the raw reversal — and the frozen rules ruled that out too, for the same reason as the guard: which refits converged had already decided which data got counted. So the gate failed on four concordant grounds — the sign, the non-convergence, and the two failed refit counts — and the claim on file is one sentence: the registered GPT-2 dissociation test failed, with a strong descriptive routing reversal.

One follow-up was authorized, with its interpretation frozen before launch: rerun the contrast with the flavor-free patch version of depth on 80 of the pairs, using the intact model. It agreed in direction — −0.090, interval [−0.107, −0.073]; 53 pairs negative, 25 tied, 2 positive. At this point a tidy story suggests itself: retrieval finishes its routing early, parametric recall keeps needing attention late. A confound check is why that story never shipped.

Here is the confound. On every one of the 80 pairs, the retrieval prompt was more confident — its clean log-probability on the target beat the parametric prompt's by 1.71 nats on average, and that advantage correlates with the depth contrast (rank correlation −0.414). Retrieval prompts were also twice as long, 26 tokens against 12. Depth, in both versions, is a survival threshold: the deepest point at which a perturbation still flips the top answer. A prediction held with more margin survives more perturbation, mechanism or no mechanism. So the one warranted claim is narrow: under this instrument, on this frame, retrieval predictions were more robust to having their context scrambled — which may be routing, and may just be confidence. The design that could split those apart, matching pairs on confidence before any intervention, became its own project.

The scoop I almost stepped on

One more deflation, this one self-inflicted and caught an hour before drafting. Through the whole project I had searched for prior work under my own name for the idea — "MLP-only lens," "MLP-continuation" — and found nothing. The lab's pre-writeup checklist forces one more search, this time in the field's vocabulary for the thing being measured: "prediction depth," "saturation events." Three direct predecessors surfaced at once. The tuned lens paper itself defines a per-token prediction depth. Geva and colleagues had described the same last-layer-where-it-fixes shape in 2022 and called it a saturation event. A 2024 paper had extended it to top-k. My planned framing — we introduce per-token prediction depth — was gone.

The collision turned out to sharpen the paper. Every one of those predecessors reads depth off a projection: a probe, trained or not, applied to an intermediate state. This one defines depth by an intervention — an actual ablated forward pass whose endpoints you can verify against the intact model. The paper's contribution became the audit between those two families of ruler, and the flip and reversal above are exactly what the audit found: constructions that were supposed to be interchangeable measurements of one underlying quantity are not.

What this doesn't settle

Why the flip happens is open. The tidy story — Qwen parks a huge standing bias in attention's output, zeroing it destroys the residual scale — made a specific prediction: the self-only flavor, which keeps each token's own attention contribution, should fail like zero does. It didn't; it landed in between. A layer-local association (rank correlation 0.69 between where the mean helps most and where the flip concentrates) points somewhere real but identifies nothing. The confidence-versus-routing question is unresolved by design — this instrument measures robustness, and matched-confidence pairs are the successor project's problem. And everything here lives on small open models, 117M to 1.4B parameters; two of the three validity cells share a corpus with the phase before them; nothing about utility, scaling, or larger families was run at all, because the gate failure canceled those phases before they spent a dollar.

The instrument is validated, cheap, and honest about its support. But what I actually trust most from these five days is not any single number. It is the pattern: every time a number looked like a property of the model, one more control found the instrument's fingerprints on it.

Most of the science ran on a laptop CPU over five days; the $13.12 above went to GPUs for Pythia-1.4B, the control cells, and the patch frames. The full write-up (a 5,857-word manuscript with supplement) is headed for journal review. References: nostalgebraist, Interpreting GPT: the logit lens (2020); Belrose et al., Eliciting latent predictions with the tuned lens (2023); Geva et al., Transformer feed-forward layers build predictions by promoting concepts (2022) and the saturation-events analysis; Baldock et al., Deep learning through the lens of example difficulty (2021); Lioubashevski et al. (2024) on top-k saturation; Queipo-de-Llano et al., Mix–Compress–Refine (2026) on depth stages.