Does a transformer snap between modes of computation?
An averaged Jacobian compresses everything above a transformer layer into one linear map. That is useful, but ambiguous: the average could summarize a few recurring computations, or many maps that change gradually with context. I looked for the first case. No hard-clustering setup passed the held-out prediction test. The local maps do vary predictably; I did not find hard seams that split them into a small number of modes.
What a Jacobian lens is
Everything above layer ℓ is a nonlinear function from the residual-stream state h_ℓ to the final state h_L. Near one particular input, its JacobianA = ∂h_L / ∂h_ℓ is the linear map that best approximates that function. For a small change δ at layer ℓ, Aδ estimates the change at the end of the model. Anthropic introduced the Jacobian lens in a July 2026 paper (Gurnee et al., 2026).
There is a different A for every prompt and token position. Attention, normalization, and MLPs all respond to context. The published lens averages these local maps into one map per layer. That average cannot tell whether the maps fall into a few groups or vary without clear boundaries. To tell, I had to inspect the variation the average throws away.
One map, a few maps, or a moving map?
Call the number of maps in a hard-cluster model K. The averaged lens is the K = 1 summary: one map for every context. At the other extreme, exact local maps (Golden, 2025) give K = N, a fresh map for every example and little simplification. I was interested in the middle: perhaps a small K could capture a handful of recurring computations.
In that picture, the local Jacobians would form tight groups: map A₁ when the layer copies the current token,A₂ when it retrieves a fact, and A₃ when it plans ahead, with context deciding which one applies. The alternative is a cloud with no hard boundaries, better predicted by a map that changes continuously with its input. The experiment compares those two descriptions; it does not assume that either one must be the model's true mechanism.
A; the published lens summarizes them with one average (left). That average cannot distinguish a few hard groups (middle) from continuous variation (right). The predictive test below asks which description works better.A cluster is not yet a regime
Ask k-means for K groups and it will return K groups, even when the data are one continuous cloud. A tidy cluster plot therefore proves very little. A useful regime needs a consequence that a continuous model would not capture.
I tested that with cross-view prediction. For each prompt, source position, and layer, the local Jacobian Aᵢ described how a small change at that point would affect the model's final residual state, averaged over valid target positions. A full Qwen Jacobian is 2560 × 2560, so I measured it through two rank-64 projections. Here R and L are fixed random probe matrices: Xᵢ = AᵢR follows 64 perturbation directions forward, while Yᵢ = AᵢᵀL traces 64 output probes backward. These are two views of the same map, not literal halves.
I put whole prompts into five folds, so every position from a held-out prompt stayed out of training. Then I predicted Y from X and ran the prediction in reverse. The score was R²: 0 means no better than the training-fold average;1 means a perfect prediction.
The hard-cluster pipeline first applies kernel PCA, a nonlinear compression step, then runs k-means and predicts the target view with that cluster's training-set average. Its comparator is rank-64 principal-component regression, or PCR. PCR reduces the source view to its 64 main directions of variation, then fits a linear map from those 64 coordinates to the target view. It has no groups: nearby coordinates produce nearby predictions.
I fixed the gate before running the clustering. A candidate had to beat the stronger of PCR-64 and an activation-nearest-neighbor control, which uses the target view from the most similar training activation. The lower end of a paired 95% prompt-bootstrap interval had to be above zero. The same lower-bound rule applied to a same-K control that clustered ordinary residual activations instead of Jacobians, using the same layer and folds. This checks whether the Jacobian itself adds information beyond grouping similar hidden states. Finally, every cluster had to receive at least 5% of the held-out rows in every fold. That last rule rejects a supposed mode that appears only as a tiny, unstable group.
The comparison is ΔR² = R²cluster − R²control. An interval such as [0.01, 0.07] passes. An interval such as [-0.02, 0.08] does not, even though its point estimate favors the cluster. The bootstrap resampled prompts 500 times; it did not refit the models or correct across all tested configurations.
The clusters lost
The main Qwen3-4B bank contained 270 prompts, up to two source positions per prompt, and nine layers. One short prompt supplied only one valid position, so the total was (269 × 2 + 1) × 9 = 4,851 cases. At each layer I tested the 15 values of K from 2 through 16 in both prediction directions: 9 × 15 × 2 = 270 configurations. None cleared the gate. Six beat PCR on the point estimate, none had a positive 95% lower bound, and only nine kept every cluster above the 5% utilization floor.
I checked the result two more ways. First I made fresh prompt folds and split each 64-column projection into two 32-column blocks. Predicting from each block to the other produced four directions and 540 new configurations. Eight had positive lower bounds over PCR, all at layer 4 with high K; every one failed the utilization rule. Then I moved to GPT-2-small, where I could compute the exact 768 × 768 Jacobian. This checked whether Qwen's projection had erased a cluster boundary. Across 360 dense configurations, no hard cluster beat PCR even on the point estimate. In all 24 layer-and-view cells, the ordering was PCR, Jacobian clusters, activation clusters, then the global mean.
K for that layer and direction, a choice that favors the cluster model, then subtracts PCR-64's held-out cross-view R². Every 95% prompt-bootstrap interval lies below zero. Across the full sweep, 0 of 270 configurations cleared all three parts of the gate.What the average missed
The cluster result does not make K = 1 sufficient. The global-mean predictor has R² = 0 by construction, while PCR averaged 0.463 across the ten Qwen layer-and-direction combinations. One projected view predicted the other on prompts left out of fitting. The local maps therefore contain predictable variation that one average discards. PCR winning does not prove that this variation lies on a smooth manifold; it shows that the tested hard conditional means lost useful information.
There was a simpler explanation to rule out. Suppose later token positions make both projections larger. Then X could predict Y mainly because it reveals token position, not because it captures richer Jacobian structure. Prompt length, token frequency, residual norm, and model uncertainty could create the same problem.
I fit a ridge model using source position, token frequency, residual norm, top-logit entropy, prompt length, task family, and data-collection shard. On the same held-out rows and folds, it reached R² = 0.207, or 44.7% of PCR's 0.463. The paired difference was +0.256, with a 95% interval of [0.212, 0.300], and stayed positive at every registered layer. This is a difference between two predictors' scores, not a literal residualization or variance decomposition. The tested metadata did not account for all of PCR's performance; nonlinear or unmeasured nuisance features remain possible.
+0.256 score difference has a 95% prompt-bootstrap interval of [0.212, 0.300].The pattern I was looking for
The toggle makes the comparison visible. In the hypothetical panel, the best hard-cluster model sits above PCR-64: knowing the group adds predictive power that continuous regression misses. In the measured panel, the cluster model stays below PCR-64 at every registered layer.
R² across five Qwen layers, view direction A R → Aᵀ L. The global mean (K = 1) sits at zero by construction. In the measured panel the hard-cluster model stays below PCR-64 at every layer. The hypothetical panel is illustrative: it shows the pattern a passing result would have made.The clusters lost, but the global average lost too. The defensible result is narrower: these projected local Jacobians contain held-out, context-dependent variation. The tested K = 2…16 kernel-PCA/k-means models predict it worse than PCR-64.
PCR-64 is deliberately strong: it has more freedom than a small-K cluster model, so this is a strict prediction test rather than a parameter-matched contest. The Qwen prompt bank is synthetic and template-heavy; the GPT-2 prompts are a verbatim subset. Because the prediction gate failed, I did not fit regime-conditioned lenses or run interventions. The result is predictive; the causal question remains open. This experiment does not show that transformers never use discrete modes or that K = 1 is enough. It also does not establish that the Jacobians lie on a smooth low-dimensional manifold.
The regeneration package is at github.com/Ian2x/jacobian-hard-clustering. For a companion negative result, see the robust-optimization writeup; for how these experiments are run, see how the lab works.