The query decides whether a repaired KV cache is safe
Language-model serving has a blunt rule about reuse: a saved cache is only good for a prompt that matches it exactly, token for token. Edit one word in a 4,000-token document and the model reads the whole thing again. There is a well-known middle path — keep the cache, recompute just the edited span, splice — and after an equal-length edit the splice is exact everywhere except for the tokens that come after the edit. I spent nine days measuring when that leftover staleness matters. Across two studies, the spliced cache never once changed the model's answer to a question unrelated to the edit: 0 of 252, both times. On questions that read the edited fact, it changed the answer about a third of the time. Which way a case goes depends far less on the edit than on the question you ask afterward. So I tried to build the gate you would actually want: one that looks at the question and predicts whether the splice will hold. The prediction works. What failed — twice, once on compute and once on calibration — was turning it into a method, and mapping those two failures is most of what this post is about.
The cache you throw away
When a transformer reads a prompt, every token leaves something behind for later tokens to consult: at each layer, a key vector (roughly, what this token is about) and a value vector (what it contributes if attended to). Attention is later tokens looking back over those stored pairs. The store is the KV cache, and building it (the prefill) is the expensive part of handling a long prompt. Answering a short question about a long document is mostly the cost of reading the document.
Caches are reusable at all because of a one-way street in the architecture: attention only looks backward, so a token's cached state depends on its position and the tokens before it, and on nothing after. Same prefix, same states. That is also why reuse is brittle. The moment one token differs, every state downstream of it was computed while reading a different document, and the strict remedy is to throw all of it away. Serving systems do exactly that: exact prefix match or full recompute.
Near misses are everywhere. Version 2 of a contract differs from version 1 by a date; a retrieval system serves the same page with one sentence updated. Throwing away 99% of a cache because 1% of the text changed is wasteful enough that people have built patches for it.
Two exact pieces and one stale one
Restrict to the friendliest edit there is: a substitution with the same number of tokens, so every unedited token keeps its exact position. Then a patched cache has three parts. Everything before the edit can be kept as is — those states never saw the edit, so they are the new document's states, exactly. The edited span gets re-encoded: run the model over the new tokens on top of that shared prefix, a small prefill, exact by construction. The problem is everything after the edit. Those states were computed while reading the old span, and the splice keeps them anyway. That is the entire approximation: stale post-edit states, nothing else. The project calls it drift.
"Exact" means exact in real arithmetic; floating point adds wobble on top. Before any behavior was measured, the splice pipeline had to pass a frozen numerical contract: identity checks, tolerance bounds, and a deliberately corrupted control that must fail loudly, so a silent bug in the splice couldn't masquerade as an interesting finding. The details are in the fold.
The numerical contract, briefly
Reusing the cache with no edit at all had to reproduce the model's greedy output — the answer it gives when it always takes its single most likely next token, so any change shows — bit for bit (it did, on every case, on CPU and on Apple's MPS backend). Re-encoding a span had to match a full prefill within a preregistered tolerance. The original absolute bound of 5×10−4 proved too tight for the real document shapes — the first checks at those shapes exceeded it even at 2K tokens — so the advisor approved a recorded amendment to 5×10−3 for all lengths through 8K, with the relative bound held at 10−4. Under the amended bounds the worst valid case came in around 1.2×10−3 absolute and 1.8×10−5 relative. Recomputing that worst case in 64-bit floats shrank the error by a factor of about 4×108, pinning it on float32 accumulation rather than a logic bug. A deliberately corrupted cache overshot the original bound by 9,600× and the amended one by 631×, so the contract detects real damage either way. And across all 630 paired checks between CPU and Apple-MPS runs, greedy outputs never diverged.
None of these ingredients are mine, and the study claims none of them. Splice-style repair is prior art (CacheBlend is the well-known version); 2025–26 produced a whole line of systems that gate cache reuse on the query; one recent paper even pilots a rough safe-or-unsafe predictor for cache reuse in general. What I couldn't find was a controlled version of the underlying question — same document, one aligned edit, the query held fixed: does the drift change this answer, at a promised error rate? That became the project. Characterize when the stale tail flips the output, then test whether a probe — a small classifier watching the model for signs of trouble — can predict it.
What counts as a catastrophe
Real document pairs conflate a dozen variables, so the workload is synthetic and deliberately controlled: template documents with a specific fact planted in filler, each paired with an edited twin, generated by deterministic code so every case is reproducible byte for byte. Seven edit families run from harmless to nasty: churned boilerplate, swapped synonyms, swapped entities, changed numbers and dates, negated sentences, a rewritten answer-bearing sentence, and — the family that ends up mattering most — an edit whose value a later summary sentence carries forward. Edits land 10%, 50%, or 90% of the way into documents of 2K to 8K tokens.
Each document pair gets three kinds of question. An unrelated query asks about a second fact each document plants elsewhere, untouched by the edit. A direct query reads or applies the edited fact itself. A multi-hop query needs the edited value plus a rule stated elsewhere in the document. Every question has a short answer from a fixed set of labels, so grading is mechanical. The failure label is blunt: run the query once with the spliced cache and once with a full recompute, and call it a catastrophe when the extracted answers differ. The answers have to differ outright; drifted probabilities don't count. (Occasionally the spliced run's answer isn't even on the menu: one banked case answered ACTIVE when the allowed labels were POSITIVE and NEGATIVE. That counts too.) Whenever this post says an answer was "wrong," it means exactly this: it disagreed with the full recompute.
One filter matters for the arithmetic: if the model gets a question wrong even with a fresh, fully recomputed cache, a mismatch there says nothing about caching; the model simply can't do that case. So everything is scored only on competent examples, the ones the full recompute answers correctly. In study 1 that kept 623 of 756 test queries, and the cut is reported rather than hidden. Study 1 covered 630 generated scenarios and 3,780 queries in all, with training, calibration, and test splits frozen before scoring.
The bystanders never break
Score the spliced cache first, no gating. On queries unrelated to the edit, drift never changed an answer: 0 of 252 competent cases in study 1, and 0 of 252 again on completely fresh data in study 2. (Zero observed is not zero guaranteed — at this sample size an exact 95% interval still allows a rate up to about 1.5% — but it never happened, twice.) On queries that read the edit, directly or through a rule, the spliced cache flipped 131 of 371 competent cases in study 1: 35%.
Splicing does help, for the record. Keeping the old cache with no repair at all agreed with the full recompute on 75.3% of study-1 cases; the splice raised that to 79.0%. On study 2's fresh draw the pair was 80.4% against 84.6%. That is a real improvement, but nowhere near safe: the repair fixes the span and can't reach anything downstream of it.
The failures also cluster tightly by edit family. On study 2's fresh test set, synonym swaps and number changes never produced a catastrophe, and boilerplate churn produced exactly one, in 30 direct queries. Nearly all the damage sits in three families: negations, rewritten answer sentences, and above all the aggregator family, where a sentence after the edit restates the edited value. Aggregator edits flipped 27 of 32 competent direct queries. Entity swaps sit between the clean families and those three.
Here is one of the banked failures.
the edit the old document says FORWARD_CODE = brown; the new one says FORWARD_CODE = white.
after the edit the unedited suffix instructs the reader to carry the code into a summary, then states the rule: white maps to PRIMARY, brown maps to SECONDARY.
the query resolve FORWARD_CODE to its label.
full recompute PRIMARY — correct.
spliced cache SECONDARY — the answer for brown.
The rule sentence sits after the edit, so its cached states date from when the code was brown, and the document explicitly told the reader to carry the code forward into the summary. The cached suffix did exactly that: it carried brown. The splice put white into the span, but the carried copy living in the suffix states still says brown, and the model answers from the carried copy. The study is careful about the status of this story: nobody intervened on the carrier sentence directly, so it's a consistent reading, not a proven mechanism. But the family-level pattern leans hard in its direction: the edits that break the cache are the ones whose value gets restated downstream.
negation the edit flipped ACCESS_STATE from inactive to active. Asked to resolve the state, the full recompute says ACTIVE; the spliced cache says INACTIVE. (This one is banked from study 1; the two below, like the aggregator case above, are study 2.)
changed answer the answer-bearing sentence itself was rewritten — the disposition changed from lost to won — and a note and rule after it still carry the old value. Recompute: POSITIVE. Spliced cache: NEGATIVE.
entity swap the owner changed from Ethan to Nora; a later line says to preserve the owner field exactly, and an owner rule maps Nora to PRIMARY, Ethan to SECONDARY. Recompute: PRIMARY. Spliced cache: SECONDARY.
Look at where the damage lives in the changed-answer case. The rewritten sentence itself sits inside the re-encoded span, so its states are exact; what answers wrongly is the note after it, still carrying the old value in its stale states. Even the entity swap fails through a later line that references the edited field — the same route again.
Seeing it coming
The obvious response is a gate: look at the case, then either accept the cached answer or pay for the recompute. Two numbers score a gate. Coverage is the fraction of competent cases it accepts. Selective risk is the fraction of accepted cases where the spliced answer was wrong. Fix a risk budget (1% throughout this project), tune the gate's threshold to hit that budget on calibration data it never trained on, and the question becomes coverage: how much recomputing does the gate save at the error rate it promised?
Each gate is a small tree-based classifier — cheap and unglamorous, no neural network. The only thing that varies is what it's allowed to look at:
- Edit only. Sees the edit and nothing else — no query. A control: if this works, the whole query-conditioning story collapses.
- Text + shallow. Cheap text features of the edit and the query, plus activations from the model's first few layers. This is the deployable baseline: everything it reads is affordable at serving time.
- Trajectory. The tier this project was really about. Run the query against the spliced cache, and while the model answers, record where each layer's heads put their attention — 2,016 numbers summarizing how much the run leaned on the re-encoded span versus the stale suffix. Then decide whether to trust the answer it just produced. A twin probe does the same thing against the old, unrepaired cache and rides along as a control.
- Oracle. Sees everything, including the fully recomputed target cache. Deliberately unfair — computing its features costs the very recompute the gate exists to avoid — so it's the ceiling to compare against, not a method.
At the 1% budget in study 1, the edit-only control covered 1.3% of cases, and the oracle covered 65.7% at 0.5% risk — a 64-point spread, the largest effect in the project. Knowing the edit without the question is nearly worthless. The text + shallow baseline landed at 56.5% coverage, though its realized risk came in at 1.7%. Against the preregistered bar — the oracle gate had to beat that baseline by ten coverage points — the measured gap was 9.15, with a resampled 95% interval from 5.3 to 13.2. Under this lab's rules a result within two points of a bar is neither rounded up nor quietly failed; it goes to a pre-named adjudicator, who passed this one. The number on file is 9.15, "adjudicated pass."
The tier worth staring at is the trajectory probe. In study 1 it tracked close behind the oracle with no access to the recomputed target — an after-the-fact diagnostic there, which is why it isn't in the study-1 panel above and why study 2 exists. It looked like the method: it conditions on the query, it nearly matches the oracle, and a serving system could run the query pass anyway and gate on what it sees. What remained was the price.
The gate that cost more than the door
A gate that saves recomputes has a compute budget of its own. The preregistered rule: a probe's added latency — counting the span repair, the probe's own pass, and the query it serves, but not the old cache's prefill, which is already paid for — must stay under 20% of the full recompute it replaces, measured on the serving GPU, in every context-length and edit-depth cell, no averaging away a bad one. None of the thirteen candidate probes passed — the bench ran from 2K to 8K contexts, and every candidate blew the ceiling in its 2K cell. The best, the trajectory probe on the spliced cache, cost 24.6% of a full recompute there. The worst, a probe that read per-head attention patterns to hunt for carrier behavior, cost 10.4 times the recompute it was supposed to avoid.
The same measurements pointed at the way out. A probe's cost is roughly a fixed query pass, while prefill grows much faster with context, so the ratio collapses as documents get longer: 24.6% of a recompute at 2K becomes 11.3% at 4K and 5.2% at 8K. At 2K the study was dead under its own rule, and the rule was the right one — a gate that costs a quarter of what it saves is not a gate. So the project preregistered a second study where the economics work: fresh data, 4K and 8K only, and the candidate probe to be picked by a selection rule already on file.
The rematch at 4K and 8K
Study 2 was designed so the second chance couldn't be an easier test. New generator seeds with zero overlap with study 1; a sealed test set that stayed sealed until every threshold was frozen; and the deployable candidate — the trajectory probe on the spliced cache — selected from study-1 development metrics before any new data was opened. The compute ceiling was re-measured first and passed in all six cells, worst case 11.1%. Then came the comparison. At the 1% budget the trajectory probe covered 77.7% of cases against the baseline's 69.5%: a gain of 8.2 points, interval 5.1 to 11.3, and this time prospective and confirmatory rather than diagnostic.
That part held. The registered endpoint asked for something stronger: recover at least half of the coverage gap between the baseline and the oracle ceiling. The point estimate cleared it, at 64.6% of the gap. But the margin over the halfway mark was +1.85 points with an interval from −0.4 to +4.1 — inside the same two-point band that had sent study 1's 9.15 to adjudication. This time the ruling went the other way: endpoint not established. The secondary checks backed that caution up. At a looser 5% budget the probe recovered only 12.5% of the gap; re-comparing the probes at matched risk shrank the margin to +0.7 points; and under the study's alternative competence rule, the same matched-risk check put the probe slightly below the baseline. The write-up on file records that adverse direction explicitly.
The quieter failure is the one that closed the method. Every gate's threshold was tuned on calibration data to land at 1% risk, and every gate hit that mark there — each one, as it happens, accepting exactly four calibration catastrophes. Applied unchanged to the sealed test set, almost every gate came in above its promise:
| gate (all calibrated to 1%) | risk on calibration data | risk on the sealed test |
|---|---|---|
| text + shallow baseline | 0.90% | 1.62% |
| trajectory, spliced cache | 0.84% | 2.28% |
| trajectory, unrepaired cache | 0.90% | 0.90% |
| oracle ceiling | 0.82% | 4.11% |
(The one gate that kept its promise, curiously, was the control watching the unrepaired cache — at lower coverage.) The study can't cleanly split how much of that inflation is finite-sample noise and how much is real shift between the calibration and test draws; at this sample size the two look alike. And the threshold rule guarantees nothing on its own: it picks the largest threshold whose risk fits the budget on the calibration draw, with no finite-sample certificate behind it. A follow-up analysis measured what a real certificate would take — thousands of calibration groups, when the study had 126. So the probe still ranks failures well, but a method that states 1% and delivers 2.3% has broken its one promise, and the deployable claim ends there. Freezing the thresholds before the test set was opened is what made the break visible instead of quietly absorbed.
On one model and one controlled workload, an equal-length cache splice leaves a failure regime that is real and sharply query-conditioned: zero flipped answers on unrelated queries in both studies, 35% on edit-reading queries in the first, concentrated where a downstream sentence carries the edited value. A probe that watches the query run against the spliced cache ranks those failures well: a preregistered 8.2-point coverage win over a shallow baseline at 4K–8K, under a 20% compute ceiling it passes. What the project could not establish, by its own frozen rules, is the method: the half-the-gap recovery endpoint landed inside the adjudication band, and the 1%-calibrated gate delivered 2.28% on fresh data. A characterization, not a product.
What this doesn't settle
Everything above is one model (Qwen2.5-1.5B-Instruct), one synthetic generator, and the gentlest edit type in the study's scope: equal-length substitutions. Study 2's fresh draw reused the same seven template families with new surface details, so "generalizes across document styles" was never tested, let alone across models. Insertions and deletions — the edits that shift token positions — are out of scope, and so is anything but full causal attention. The graded answers are two or three tokens, so nothing here speaks to long generations degrading in subtler ways. The carrier story remains a reading of the pattern; the intervention that would pin it down (edit the carrier sentence itself, hold everything else fixed) was designed but never run. And the trajectory probe's win over the baseline is a ranking of separately trained predictors; it does not prove the trajectory contains information the baseline lacks.
Two smaller disclosures the project's rules require in any write-up. In study 1, one cell was underpowered: the model was competent on only 6 of 36 changed-answer direct queries, so that family's study-1 numbers rest on very few cases. And an early bug in the answer-extraction code was found and fixed mid-project; an audit re-ran the extraction across all 11,340 recorded generations and found zero labels changed by the fix.
I went in to build a gate and came out with a map. Unrelated queries never broke, and edit-reading queries broke a third of the time; telling the two apart cheaply, at a promised error rate, is what this attempt couldn't do. The manuscript's summary is the sentence I'd defend anywhere: these results establish a query-conditioned failure regime, not a general cache-reuse method. The next attempt has two walls to clear — a probe cheap enough below 4K, and a calibration that survives fresh data.
Model: Qwen2.5-1.5B-Instruct in float32 with eager (plain, unoptimized) attention; model timing and collection on a single A10G GPU, scoring and gate-freezing on cloud CPU, orchestration from a laptop. Every number and every plotted point traces to the project's frozen, hash-pinned result artifacts; the figures here are redrawn from the same files the manuscript renders from. A full manuscript in TMLR format is written and in final audit before submission. Prior art the study builds on and claims no part of: CacheBlend (Yao et al., 2024) for splice-style repair; ProphetKV, QCFuse, KVEraser, Leyline, and Models Take Notes at Prefill (2025–26) for query-aware reuse and selective recomputation; When KV Cache Reuse Fails in Multi-Agent Systems (2026) for a first generic safe-unsafe reuse predictor; the risk–coverage framing comes from the selective-prediction literature.