Does model editing store facts, or just answers?

July 11, 2026 · research by AI agents, directed by me

A model editor makes a persuasive demo. Tell a frozen model that the Eiffel Tower is now in Rome, ask the question several different ways, and watch it keep saying Rome. This is called transfer. It is usually taken as evidence that the edit changed something more durable than one memorized prompt. But the score only sees the answer. It cannot tell whether the model changed an Eiffel-Tower/location association or merely became unusually eager to say “Rome.”

You could force the answer at the very end. A transformer finishes each step by turning its working vector into a score for every word in its vocabulary, then saying whichever word scores highest. Reach in and add to the score for “Rome,” and the model says Rome. That is a boring trick, and it settles nothing: of course you can make a model say a word by pushing on that word directly.

The useful test is stricter. I build what I'll call an answer-writer: a nudge that knows only the target word, forced to play by the editor's rules. Same layer, same spot in the sentence, same size of nudge, and no look at the held-out paraphrases the editor is scored on. Then I put the answer-writer and the editor on the same transfer score and watch which one wins. Sometimes the editor pulls clear. Sometimes the cheap answer-writer keeps right up. That gap, editor minus answer-writer, is the whole story.

What the score actually sees

The score I keep calling transfer has a narrow definition. After an edit, the model is asked for the new answer in wordings the editor never trained on, and it has to generate the reply one token at a time from its own words, always taking the most likely next token. Two benchmarks supply the edits. CounterFact is a large set of factual swaps, like moving the Eiffel Tower to Rome; Bias-in-Bios hands the model a short biography and asks for the person's job. The answer counts as a hit if the new target turns up in the first several generated tokens (eight for CounterFact, twelve for the longer job titles).

Generating the answer for real is a harsher test than the way editing methods are usually scored. Many benchmarks read off probabilities instead. They might call an edit a success when the new answer merely outranks the old one, even if some third word still beats both. Or they use teacher forcing: to score a multi-word answer, they feed the model the correct earlier tokens for free and check only the later ones, so it gets credit for a continuation it would never have reached on its own. Yang and colleagues showed that tricks like these explain much of the gap between glowing editing scores and what models actually say out loud (Yang et al., 2025). I score everything here the hard way, by letting the model generate.

Three editors, three places

I put three editors through this. They work in very different places inside the model, and that turns out to matter more than almost anything else about them. SAKE waits until the very end and adjusts the model's last internal state, right before it picks a word. REMEDI reaches in earlier, adding a learned vector onto the subject the model just read (the “Eiffel Tower” part of the prompt). ROME goes furthest: it edits an actual weight inside one layer (a rank-one tweak), so the new fact is baked in before any prompt even arrives. Hold that spread in mind. An answer-writer that just shoves a word onto the output should have an easy time filling in for SAKE, and a hard time filling in for an edit buried deep in the subject.

Passing the test isn't the same as being right

Here is the gap the transfer score can't see. Every paraphrase asks for the same answer, so a model that has simply grown fond of the word “Rome” will ace all of them. The trouble shows up only when you ask something else about the same subject. Say an edit is meant to change one fact about the Bank of England, and the new answer is “Hamburg.” Now ask a different question about the same bank, the year it was founded. If the model still blurts “Hamburg,” the edit wasn't specific: it didn't learn a fact, it just got stuck on a word.

This is a different check from the one editors usually report. CounterFact ships with “neighborhood” questions about other subjects, testing whether an edit bled onto unrelated facts. The check I care about here is the mirror image: same subject, different question, watching whether the new answer leaks where it shouldn't. I built it from CounterFact's own records, and the coverage is thin. The ROME set, for one, had only three genuine same-subject rows next to 147 filler prompts, so I read it as a rough gauge of whether an edit leaves a sticky answer, not a clean measurement.

The cheapest answer-writer

Start with the laziest version. Recall that the model ends each step by scoring every word in its vocabulary and keeping the winner. Those scores come out of one big table, the output matrix, which has a row for every word. The row for the target word points in the exact direction that, added to the working vector, lifts that word's score. Call that row W_U[o*], with o* standing in for the target word. Add it right at the end and the target's score rises, no subtlety to it. The cheapest answer-writer takes that same row and injects it not at the end but wherever the editor works, deep inside the model. It knows the answer and nothing else.

One rule keeps the contest fair: the answer-writer's nudge has to be the same length as the editor's. Length here is just the ordinary geometric size of the vector, the thing I will keep calling its norm. Spelled out, if the editor adds a change I will call Δeditor, and the answer-writer pushes along some direction v, then it adds:

That is all the equation says: aim the push at the target word, then stretch or shrink it until it is exactly as long as the editor's own change. The length rule matters because a bigger nudge is a bigger effect, and without it the answer-writer could win just by shoving harder. Equal length is not equal effect, though. Two pushes of the same size can be magnified very differently by the layers they still have to travel through.

The same direction, two very different results

Here is the catch that makes the whole exercise interesting. That output row acts like a clean score bump only when you add it at the very end. It is just a vector, the same shape as the working vector, so nothing stops you from adding it earlier instead. But do that and it stops being a score bump. Now it has to survive every layer that still comes after it. And if you add it onto the subject token rather than the last one, its effect also has to move sideways, from the subject across to the final position, because that final position is the one that actually picks the next word.

GPT-J makes the point starkly. Add that same norm-matched output row at layer 1 and score the paraphrases: on the final prompt token it lands at 1.000, and on the subject token it drops to 0.103. Same direction, same length, same early layer, and roughly 26 more layers to go in both cases. So the push has no trouble riding straight up through the layers to the output. What kills it is the sideways trip. The moment it has to travel from the subject token over to the final position, it nearly vanishes.

There is a second trap, and it is a nice one. GPT-J happens to keep two separate tables: one for reading a word in, one for writing a word out. They are not the same table. So the row that means “I just read the word Rome” is not the row that means “say Rome next.” Use the reading row, W_E[o*], at the final position and it scores a flat 0.000. Not small, not noisy: zero. It is a genuine null, exactly what you want a control to do when it truly carries nothing useful. For contrast, REMEDI's own learned vector, added at that same early subject site, reaches 0.746. So a bare answer row is a weak stand-in this deep in the model, which is the reason the contest hands the answer-writer better tools further down.

Giving the answer-writer more to work with

The bare output row is a fair stand-in right at SAKE's output-facing site and a poor one deep in the model, so a single verdict from it would not be fair to every editor. The fix is a ladder. It is the same answer-writer all the way up, always pinned to the editor's own site, token position, and nudge length. The only thing that grows is how much it is allowed to learn about the frozen model it has to push through. Not one rung ever sees the held-out paraphrases it will be graded on.

1. Rung 1, the bare answer. Drop the output row W_U[o*] in at the editor's site. Where the model keeps separate read and write tables, also try the reading row W_E[o*]. This rung knows the target word and nothing else.
2. Rung 2, one step downhill. Let the answer-writer peek at the frozen model just enough to ask which direction, right here, most lowers the error on the target word, and push that way. It is the best move for one tiny step. Pushed the full length, it can be beaten, because the model bends as you go.
3. Rung 3, the tuned answer. Start from that downhill direction and keep going: adjust one vector over several steps to make the model say the target word on the rewrite prompt, resetting it to the editor's length after each step. This is the strongest answer-writer I build, and even it only ever sees the one rewrite prompt, never the paraphrases it is scored on.
4. Rung 3.5, the same answer in many contexts. Tune one shared vector on the rewrite dropped into 21 everyday sentences, averaging the error across all of them, rather than fitting 21 separate vectors. This was a planned stress test of how far the idea scales. Its one finished run tripped a validation check, so it stays on the chart as a hollow point, not a result.

So calling the top rungs “fact-free” would be a stretch: they have studied the rewrite prompt and the target word. The honesty is in the leash. Each one is still a single vector, held to the editor's length, tuned without ever seeing a paraphrase, and made to reach the output the same way the editor does. If that constrained little push can match the editor across fresh wordings, the editor's score never needed a fact behind it.

One number carries the whole contest. For each edit I take the editor's transfer score and subtract the best score any answer-writer reached on that same surface, and I call the gap the residual. Negative means some answer-writer already kept up, so the editor earned no real credit. Positive means every answer-writer I tried fell short. To tell a real gap from luck, I resample the edits many times and watch how much the gap moves; the interval I report is the band it stays inside 95% of the time. Because the editor and the answer-writer face the very same prompts, I can line them up edit by edit, which tightens that band. A gap counts only when the whole band sits off zero.

The logic runs one way only. A negative residual is a clean verdict: something as cheap as writing the answer already reproduces the score, so the score proved nothing about facts. A positive residual is softer. It says my particular answer-writer could not fake the score, not that no shortcut ever could, and it never tells you what the editor did instead. Failing to fake a score is not proof the score was real.

The ladder gives different answers

From here on the results come one surface at a time. A surface is a single fixed setup: this editor, on this model, doing this task, at this spot, scored this way. Two of them show up below. CounterFact swaps a fact in a short statement, turning “Circuit des Ardennes is located in Belgium” into “Russia.” Bias-in-Bios asks a different question: read a one-line biography, pick the job out of 28 options, and make the person's name pull up that job. The two run on different answers, layers, and REMEDI editors, so it would be meaningless to average their residuals into one score.

Reading the chart: the dashed line is the editor's own transfer score, each bar is one rung of the answer-writer on that same surface, and the gap between them is the residual.

SAKE is the easy case, and the answer-writer wins it. SAKE works right at the output, the one place a bare output row should be strong. It scores 0.895 on transfer, and the norm-matched bare row hits a perfect 1.000. So the cheap trick doesn't just tie SAKE here, it edges ahead, and the interval stays clear of zero: the residual is −0.105, spanning [−0.140, −0.075]. That is not the end of SAKE's story yet. On the same-subject leakage check it is the more careful of the two, emitting the target where it shouldn't only 0.680 of the time against the answer-writer's 1.000. The bare row wins the transfer contest and loses the leakage contest.

ROME is where the story turns on which answer-writer you use. ROME lives deep in the weights, the kind of spot where bare output pushes go to die. Sure enough, the bare row manages only 0.02 against ROME's own 0.68, and one step downhill lifts it to just 0.09. Let the answer-writer tune itself, though, and it climbs to 0.52, a residual of +0.16 with interval [0.09, 0.24] that still leaves ROME ahead, but nothing like the runaway win the bare row alone would suggest.

REMEDI on CounterFact is the one editor that holds its ground. Here the tuned answer-writer, the strongest one I have, still cannot catch up. REMEDI scores 0.706; the answer-writer stalls at 0.296, a residual of +0.410 that sits far from zero. This is the most interesting result in the audit, because it is the one place where an editor did something a same-site answer-writer could not fake. The same gap shows up again on GPT-J (see the forest plot below), so it is not a fluke of one model. I keep the claim narrow all the same: it holds against the controls I actually ran, and the context-averaged rung never finished on either REMEDI surface.

The same editor, flipped

Move REMEDI to Bias-in-Bios and the verdict turns over. The tuned answer-writer is the winner here. REMEDI scores only 0.333; the answer-writer reaches 0.522; and the residual, −0.190 across [−0.275, −0.108], comes out negative. It is the same recipe that resisted every control on CounterFact, but the task, the layer, the answer space, and the training data have all changed. So “REMEDI passes” and “REMEDI fails” are both the wrong shape of sentence: the surface is what passes or fails, not the method.

And transfer still is not the whole story. On the same-subject leakage check for Bias-in-Bios, REMEDI emits the target where it shouldn't just 0.075 of the time, while the tuned answer-writer does it at 0.390. So the answer-writer both produces the right occupation more often and blurts it out in the wrong places more often. A good transfer residual still doesn't make something a good editor.

What this doesn't settle

Plenty. Most of the surfaces run on 50 single-token CounterFact edits on a few older models, so this is a probe, not a survey. The Qwen ROME check leans on a public port rather than an official one. My same-subject leakage check is thin, especially for ROME, and the token cutoffs that decide a hit were inherited, not tuned here. Matching norms equalizes the length of a nudge, not how hard it is to optimize or how much it disturbs the output. And rung 3.5, the context-averaged bundle, never finished, so it proves nothing on its own. I also tried to test whether a big residual predicts a genuinely useful editor; that run collapsed when the held-out editor failed to reproduce.

The habit I would keep is small. When an edit “transfers,” have something cheap write the answer from the same spot before you reach for a story about facts and mechanisms. If the cheap thing keeps up, the score was never as strong as it looked. If it falls short, say exactly which version fell short. The failure does not name the mechanism for you.

References: Meng, Bau, Andonian & Belinkov, Locating and editing factual associations in GPT (ROME, 2022); Hernandez, Li & Andreas, Inspecting and editing knowledge representations in language models (REMEDI, 2024); Scialanga et al., SAKE (2025); Yang et al., The Mirage of model editing (2025). Specificity-failure context: Cohen et al. (RippleEdits), Zhang et al. (EVOKE), Hoelscher-Obermaier et al. (CounterFact+).