Killing bad ideas early
Eleven of the lab's thirty-odd ideas are buried here. Together they cost about four dollars of cloud compute, and eight of them cost nothing at all. None of them got far enough to eat a month of work and a serious bill.
The gates
Every idea runs the same five gates, ordered by how early each one can fire. A prior-art check comes first, before any code: does someone already own this? Then arithmetic — a bound on a napkin that can rule the idea out before a single GPU turns on. Then a preregistered gate: a specific number, on a specific test, written down before the lab touches any data, that ends the project if it comes back wrong. Then a control, the dumbest thing that could work, included only to lose. Usually that is a baseline I could have shipped instead, and if it wins, the clever method earned nothing. Sometimes it is an oracle I could never deploy, and if it wins, the problem is subtler: the score cannot tell the two apart. Last, real data, where a synthetic effect either survives contact with real features or evaporates.
The gate has to be frozen before the run, or it is just a description of whatever came back. That alone does not hold — I can talk myself into anything — so the lab also keeps an adversarial reviewer whose only job is to embarrass the first result, and a lead who reads the raw output instead of the summary. One idea below died that way. On breadth-ratchet the agent running the experiment logged a confident "GO"; the lead opened the same JSON and killed it.
The board below is the graveyard. Each headstone is a dead project, sorted into the gate that killed it. Click one to read the cause of death and the number that did it.
Two of the eleven died before the real experiment was ever built. homeostatic-backward-routing lost to the literature. The plan was to send each backward pass to the units that had been getting the least of it, on the theory that starved units are wasted capacity. But the base move, routing gradients to only the top few units, is meProp, published years ago. The homeostatic twist is spread across a stack of others: BlockLLM already divides its selection score by how often a unit gets picked, and Ahmad and Scheinkman already boost neurons by duty cycle. One piece survived the audit: a feedback term on how recently a unit had been chosen. A five-seed canary on a toy problem buried that too. Adding homeostasis to plain top-k moved the error from 0.142 to 0.249 — worse than the method it was meant to improve, and worse on every task in the set.
latent-video-world-models lost to one inequality. Compressing a video predictor's attention cache leaves fewer bytes to read back each step, which should make it faster. Except the predictor decodes 196 tokens per step, which pins the cache read at 196 arithmetic operations per byte, and an A10G stops being limited by memory above about 117. The read was never the bottleneck, so shrinking it could not pay. That holds at any sequence length, because the length cancels out of the ratio. A benchmark run afterward agreed: cutting the cache eightfold bought 3%, and the fancier compression the whole idea depended on ran four times slower.
The two kills I keep coming back to are the ones where my own rules cost me something. context-keys-collide had to pass 40 of 40 setup checks before it could launch, and it came back with 39. The one failure was a stale test fixture, and my own advisor called the repair mechanical. But the retry budget was zero, written down weeks earlier, so the project died there having never run a model. triadic-residual-pilot had to prove its statistics worked before it could open a 43 GB corpus: on data engineered to contain nothing, it had to raise a false alarm on about 5% of it, which is the property that makes its p-values mean anything at all. It came in at 4.29%, missing the band by two tenths of a percentage point. Its confidence interval, 3.84% to 4.78%, still overlapped the band's floor, so I could have called the miss noise and moved on. The rule said stop, and the corpus stayed shut.
That gate had already failed once, in the other direction. Its first version was impossible to pass. A quarter of the test mixture was degenerate tables that can never raise an alarm at all, which capped the achievable rate near 3.75% against a band that started at 4.5%. No instrument could have cleared it. Finding that out cost a run, an advisor to diagnose it, and the project's one permitted repair to rewrite the gate — which is why nothing was left to spend when the rewritten version came back at 4.29%. Nobody had thought to check whether the number was reachable before freezing it. Before the gate was refrozen, a script walked every other threshold in the project and confirmed that some result could actually satisfy it.
vesuvius-unwrapping is still running, so it has no headstone, but it belongs here. It wrote itself a fail-closed rulebook and then bound itself with it. The tooling hardcoded aws_gpu_launch="blocked" in every launch gate, and the evidence ladder defined the first tier that would unlock a launch as artifacts the project had already proven do not exist publicly. Every one of those rules was a sensible default failing safe. By the end of June the project had established, by its own rules, that nothing it was permitted to run could ever pass its own criteria for scaling up. It then ran about thirty more experiments.
Only one idea reached the last gate. prior-vs-replay predicted, in closed form, when a learner should lean on a compact structure it already knows instead of hoarding old examples to retrain on. On synthetic data the prediction landed exactly. On real MNIST features it dissolved. The prediction turns on a ratio: how many dimensions the data spreads across, over how many the structure actually needs. Synthetic data let me dial that ratio up to 128. Real features are not compact in that way — the structure sprawls across dozens of dimensions instead of nine — so the same ratio comes out between 1.3 and 6.2, and the effect has no room to appear. Even that kill was free: three minutes on a CPU.
Why I stopped the fact-subspace project
I wanted to know whether a language model stores a fact like "the capital of France is Paris" as something portable: one direction in its activations that I could lift out of one prompt and drop into an unrelated one. I estimated such a direction from paraphrases of the fact, at a single layer and token position, and it worked. Transplanted, it carried the answer with it, well clear of the random and template controls.
Then I ran the control that exists to lose. Instead of my estimated direction, it injects the model's own output-embedding row for the answer token, the vector that most directly means "say Paris," at the same layer and position and scaled to the same length. It encodes no fact — it just shoves the model toward a word. It scored 0.902 against my component's 0.607 on the frozen confirmation set.
That is not proof that my component was secretly the answer direction, and I checked: geometrically it is not, overlapping that row by a cosine of 0.11 to 0.24, a few percent of its energy. The control is not perfectly fair either. It is handed the answer my method had to infer, and I matched the injected vector's length without matching what it did to the output distribution. What the loss does show is that my score was not measuring the thing I needed. A pure output shove and a genuinely stored fact look the same to it, and the shove wins. My component isn't clean either: decoded through the model's own output head, it reads out as the answer token for six of the eight capital facts, which suggests it was partly doing the same cheap thing. The rank, ablation, and relation-swap checks did not break the tie. So I stopped.
The hole in all of this is that I am grading my own gates. Every number here shows that a gate fired. None of them shows that it was right. Somewhere in these eleven there may be an idea that would have worked, stopped by a threshold I picked in an afternoon, and killing things early is exactly what guarantees I will never find out which. Eleven wrong guesses for four dollars leaves me room to keep guessing; one wrong guess for a month does not.