Every importance gate is a Bayesian prior in disguise

July 10, 2026 · research by AI agents, directed by me

An importance gate is anything that scales a model's update by how much each sample supposedly matters — a per-sample learning rate. Gate well, the hope goes, and you learn more from less data. The experiments below say that hope is misplaced in a specific, measurable way: a gate is algebraically the same thing as a Bayesian prior, so it can only spend information the system already holds. When a gate seems to pull signal out of eight noisy examples, that signal was imported from somewhere, and you can measure how much.

A trick the brain seems to use

Your brain does not learn at a constant rate. Most of today will leave no trace on you, while one startling moment can rewire you for good. Copying that into a model sounds straightforward — find the examples that matter, learn hard on those, dampen the rest — and the obvious customers are continual learning and few-shot learning, which are both short on data. So the question worth testing: is there a task family where importance-gated learning beats plain uniform training on few-shot generalization? Everything below hangs on one fact: a learning-rate gate, a Bayesian prior, and a precision are the same object.

The same knob, three names

Start with the smallest possible example. You believe some quantity sits at m = 0, and a fresh reading comes in at y = 1. A gate writes the update as a delta rule, m′ = m + g·(y − m), where the learning rate g, between 0 and 1, says how much this observation deserves to move you.

A Bayesian never picks a rate; they state beliefs. Your belief is a Gaussian at m with precision τ_belief (precision is one over the variance: how sure you are). The reading is a Gaussian at y with precision τ_obs. Bayes' rule blends them into a posterior whose mean is the precision-weighted average of the two centers, which is exactly the gate's update once you set g = τ_obs / (τ_belief + τ_obs):

m′ = m + [ τ_obs / (τ_belief + τ_obs) ] · (y − m)

That fraction is the Kalman gain. An observation you trust (high τ_obs) is one you learn hard on (high g); one you don't trust, you shrink back toward what you already believed. Drag the slider and watch the learning rate and the precision weight stay equal.

Why an identity is bad news for a method

The toy was one number in one dimension, but the collapse is general. Meta-learned learning rates, gradient-agreement filters, prioritized replay, predictive-coding precision weights — write any of them as an operator that re-weights a batch of gradients, and the operator always consults something external: a base model's direction, an invariance group, a validation gradient, a model of volatility. That external thing is the prior, and the gate is just the re-weighting.

So the useful question shifts from how clever a gate is to where its prior came from, and there are only three honest ways to pay for a prior: pretrain it from past data, validate it on a held-out set, or wait and estimate it from the near future. The experiments below try each in turn.

Watching the prior get read back out

Start with the pretrained prior, in a setting where the signal is visible. In the benign-overfitting model of Cao, Chen, Belkin & Gu (2022), a learned weight decomposes into a component along a planted signal direction μ plus per-example memorization terms, so "did the update point at the signal" is a number you can read off: the cosine between the update and μ. One measurement rule matters here: every method's update is rescaled to the same norm, so direction is the only thing allowed to differ. Without that, a gate could "win" by acting like a larger learning rate, which is the easiest confound to mistake for a real effect.

The lab took a deliberately brutal few-shot gradient — eight examples at a signal-to-noise ratio of 5×10⁻³, the regime where the theory says separating signal from memorization is hardest — and projected it onto a low-rank signal subspace estimated from a base model trained on n_base fresh samples from the same distribution, sweeping n_base from 0 (random init) to 1024 (well-trained). The control is a random subspace of the same rank, rescaled to the same norm. While the base model is ignorant, the projection gains nothing: the cosine gap over the control is −0.019 at n_base = 0 and n_base = 1. Once the base model contains the signal, the gap climbs: +0.112 at n_base = 8, +0.528 at n_base = 64, +0.963 at n_base = 1024. And the alignment the gate "recovers" equals the base model's own alignment to μ almost to the digit: at n_base = 256, 0.908 versus 0.908. A label-shuffle check agrees: scramble the labels and the systematic gap collapses.

Swapping the prior changes nothing structural. The lab ran the same sweep for three gates. The base-subspace gate is the one above: it projects the update onto a subspace learned from past data, a pretrained prior. An agreement mask instead trains on eight re-noised copies of the data — views that share the signal but not the noise — and keeps only the coordinates whose sign all eight copies agree on; what it imports is the assumption about what the copies share. A validation gate computes the gradient on a small held-out set and up-weights the part of the update that points the same way; the held-out set is its prior, paid for by validation. Each gate was swept from an uninformed to an informed prior, and each first beats its matched control only once its prior has cosine alignment with μ: 0.20 for the base-subspace gate, 0.24 for the agreement mask, 0.12 for the validation gate. Plot recovered signal against prior alignment and the two directional gates land on the line y = x; the agreement mask runs below it, because it keeps a whole set of coordinates and the signal gets diluted across them.

An agreement gate discards a rare true signal

Agreement gating also has a sharper failure mode. Score importance by agreement — keep whatever many examples or augmented views concur on — and a real but rare signal gets thrown away, because a handful of examples cannot out-vote the crowd. Feldman and Zhang (2020) showed that memorizing atypical long-tail examples is necessary for optimal generalization: the deer that crossed the road once still has to be learned.

To find where the cutoff sits, the lab planted a second signal μ′, orthogonal to the main one, carried by a rare subclass that appears k times, and tracked how much of it survived the gate as k grew. Nothing survives until k* ≈ 12, the first point where retention beats its control by two standard errors and stays there. Repeating this at different dataset sizes N shows the threshold is steadiest written as k/√N: its coefficient of variation across the sweep is 0.19, versus 0.45 for a raw count and 0.48 for a frequency. That is the signature of a detection floor. The rare signal has to rise above the crowd's sampling noise, which shrinks as √N, and there is even a closed form that matches: the gate keeps a coordinate only when all eight views agree on its sign, and the probability of that unanimous vote predicts the retention curve's shape at r = 0.995. Below the threshold, the gate removes a genuine minority signal entirely.

The lab then re-ran the threshold experiment on real MNIST features. The clean k* ∝ √N law falls apart there: the fitted exponent wanders between −0.45 and 1.0 depending on the feature regime (raw pixels, PCA directions, cross-dataset digits), so the scaling claim does not survive. The threshold itself does. A genuinely rare digit class is suppressed until k* ≈ 8, and the threshold's height is set by feature covariance: plant the rare signal along a correlated top-PCA direction and the gate suppresses it up to k* = 40, against 5 for a near-isotropic direction and 4 for matched Gaussian noise, a 3σ split. Real features are correlated, so on real data the floor sits higher, not lower.

Where a gate honestly wins: volatility

A gate earns its keep only where the right learning rate changes over time. On a stationary problem a well-tuned fixed rate is already near-optimal, which leaves a gate nothing to do. So the lab went to the classic non-stationary testbed: volatile reversal learning, where a reward probability jumps at random changepoints. The hazard h is the per-step probability of a jump. There the Bayes-optimal learning rate spikes at each jump and decays through the calm stretches, a moving target no fixed rate can track.

The gate tested is the toy from the top of the page, run online. At each step it estimates the probability that a changepoint just happened, from how surprising the new observation is under current beliefs, and treats that as the observation's precision — so the learning rate jumps toward 1 at a suspected change and decays as evidence accumulates. It carries one meta-parameter, tuned once and pooled across all hazards, while both baselines, a fixed learning rate and Adam, are grid-searched separately at every hazard. The less-tuned gate still beats both at every volatile hazard, by more than four standard errors, and closes 79%, 83%, and 70% of tuned Adam's remaining gap to a Bayes-optimal oracle at h of 0.05, 0.1, and 0.2. At the band edges it earns less: 21% at h = 0.02, where the world is nearly stationary and a fixed rate is nearly right, and 49% at h = 0.35, where the world changes too fast for tracking to pay.

The useful evidence here is the shape of the curve. The gate helps most at intermediate volatility and least toward both ends — an inverted U. A gate that was secretly just a bigger step would help everywhere, monotonically; a real precision gate helps only where precision varies. (A diagnostic run that tunes the gate separately per hazard closes 93% of the gap in the mid-band, so the single pooled parameter, not the mechanism, accounts for most of the shortfall at the edges.)

The gate itself is not new. It is the reduced-Bayesian changepoint delta rule of Nassar et al. (2010), and the lab re-derived it because it marks the boundary: gating wins where a correct precision prior exists, and volatility is such a prior.

You cannot escape the prior by waiting

One escape route seems open. If eight examples cannot define what generalizes, don't decide yet: defer the update, let more examples arrive, and consolidate only the directions the future keeps confirming. On a stream whose signal keeps moving, waiting for fresh evidence should surely beat a frozen pretrained prior. The lab pre-registered the test with a binding kill switch — if waiting could not beat both a pretrained and a validated prior by two standard errors in the hardest, highest-noise settings, the idea would be declared dead — and the kill switch fired.

In this experiment, signals live in a K-dimensional subspace inside a d-dimensional space, and at each changepoint the signal moves somewhere new within that subspace. Waiting averages incoming gradients in all d dimensions, so it has to grind down the noise in the d − K useless dimensions sample by sample. Using the pretrained prior means projecting each gradient onto the K known dimensions first and then averaging: the same corroboration, confined to the right subspace, which boosts the signal-to-noise ratio by roughly d/K. Waiting done right is just the prior applied first. And in held-out loss the pretrained prior never loses — from a tie when d/K = 1, its margin grows with d/K, reaching 0.0076 ± 0.002 at d/K = 128; in the single worst cell for waiting, the margin is 0.353 ± 0.02.

This inverts the conclusion of MOCA (Harrison et al., 2020), a meta-learning method whose base learner uses the same geometry: a shared learned subspace with a per-regime readout. MOCA's claim is that a meta-learned prior lets a learner adapt quickly from post-change data. Here, in the high-noise regime, the prior dominates: no amount of post-change corroboration catches up to simply applying it.

What this shows, and what it doesn't

This has two limits. First, it is a characterization, not a method: nothing above tells you how to build a better gate, only what bounds any gate's winnings. Second, the evidence is mostly synthetic. Four of the five experiments run a linear predictor on synthetic signal-plus-noise, and the core measurement scores alignment to a planted direction the experimenter knows — an instrument that can confirm signal was imported, but cannot rule out genuine few-sample extraction on real problems. The MNIST check is the only contact with real data, and it kept the threshold while killing the scaling law. If someone shows a gate beating a matched control on a task where no prior could have been imported, this thesis is wrong. The pattern echoes the robust-optimization write-up elsewhere on this site, where "robust" always turned out to mean robust against some specific corruption — see Robust stochastic optimization: RGD vs. SEVER. Here, "important" always turns out to mean important according to some specific prior.