The most expensive 134 seconds

The filed result was “beaten by five of six baselines”. The true statement is that the forecast was the best in the table and only the confidence attached to it was wrong. Nothing in the instrument could say so, because the comparison table had no column in which the difference could appear.

The number nobody had quoted

The evaluation record carried a mean-squared-error column that no report had read. On it, our model has the lowest MSE of all seven arms.

A model can only hold the best conditional mean and nearly the worst log-score if its uncertainty is wrong. Isolating that: excess score over ½·log(2πe·MSE) — the part attributable entirely to predictive variance, given the mean — is +0.4469 for SC-WBD, against −0.1025 to −0.1249 for every statistical baseline.11An earlier internal draft quoted this baseline range as “−0.10 to −0.12”, which puts persistence marginally outside it. The corrected range is in the text. We are reproducing the correction rather than quietly using the right number, because the habit of silently fixing quoted figures is how a project loses the ability to audit itself. The deficit to persistence is 0.2765. The variance penalty is 1.62× the entire gap.

The mechanism, in two lines

The whole predictive variance for EEG:scwbd/foundation/heads.py:238

self.log_noise = nn.Parameter(torch.zeros(n_ch))   # __init__, shape (C,)
...
lv = self.log_noise.expand_as(y)                   # forward()

One learned scalar per channel, broadcast. lv never reads the state. It is constant across time, across horizon step, across window, across participant and across condition.

The module docstring claims the head learns “a heteroscedastic noise model”. It cannot: there is no path from state to variance. Meanwhile the five held-out-calibrated baselines get variance shaped (horizon, channel), so their uncertainty may grow with the forecast horizon. Ours could not. One constant covering horizons 1 through 24 is a compromise the Gaussian log-score punishes at every horizon simultaneously.

Decomposing the penalty — and the surprise

The obvious hypothesis was horizon-flatness: a variance that cannot grow with horizon, scored at 24 horizons. It was pre-registered before the measurement, and it was wrong.

Horizon-flatness is worth 0.0096 nats. It is not the cause. The cause is a scale error in a single scalar.22The decomposition procedure was validated before being trusted: run identically on all seven arms, every baseline's flat-recalibration gap is 0.0000 to four decimals — the reimplementation reproduces each baseline's own calibration exactly. A decomposition that could not reproduce a known-correct case would not be evidence about the unknown one.

The 134 seconds

The parameter eeg.log_noise has mean 0.2732 and standard deviation 0.0299 across 64 channels — flat to 3%. It asserts a variance of 1.31 when the held-out residual variance is 3.97. The model is uniformly 3.0× overconfident.

Why? The parameter was made trainable in stage V only, and stage V ran 900 steps at learning rate 5.77e-5 — 134 seconds.scwbd/foundation/train.py:78 The stationary point is log(3.97) = 1.379. It started at 0 and reached 0.273: about 20% of the way there, still drifting when training stopped.

A parameter whose optimum has a closed form was left to SGD for two minutes.

There is a second instance of the same class in the same file. The haemodynamic head's log_noise is exactly −4.0000 across all 454 regions, standard deviation exactly zero. It never received a gradient, because no measured haemodynamic data entered the corpus. Nothing was scored on it — which is why nobody noticed, and exactly why it is dangerous: the moment real data lands, an unfitted noise model is presented as a fitted one.

Model and instrument, both true

The pre-registration provided for the possibility that this was partly an instrument defect, and it was. The comparison was not calibration-matched: five of six baselines carry held-out, per-horizon, per-channel residual variance calibration. Our arm's descriptor had three fields and none of them was that.

So the instrument reported no difference because it had no field in which a difference could appear.

Recalibrated to the same standard every baseline already got, the paired participant-clustered interval against persistence is −0.2582 [−0.2868, −0.2307], excluding zero. Matched, the failure does not reproduce, and SC-WBD goes from beaten-by-five to beaten-by-none.

And on the conditional mean, with the paired intervals restored, SC-WBD beats every baseline:

This does not rescue the model, and we declined to say it did

Two refusals are the point of this section.

The failure stands. A model contracted to emit calibrated uncertainty, failing at exactly that, is a real failure and not an artifact of the instrument. The specification lists an uncertainty channel as regional state; the artifact emits a constant.

We did not promote MSE to the primary endpoint. Doing so was defensible on the merits — the intervals are real and the model wins on all of them. It was refused anyway, because selecting an endpoint after seeing the outcome is exactly what the pre-commitment was written to prevent. The temptation arrived faster than expected, and the rule held precisely because nobody was allowed to relitigate it once the numbers were visible.

Why the verification apparatus could not have caught it

The identifiability machinery found most of the other defects in this project. It could not have found this one, and the reason is structural rather than an oversight.

Those checks are exact Fisher computations on a linear-Gaussian surrogate. In that surrogate, state-independent innovation covariance is a theorem — it is precisely why the Riccati recursion can be shared across the trajectory. So a constant log_noise is correct in the surrogate. The surrogate cannot represent the defect, let alone detect it.

Every check was green and every check was correct. A verification apparatus built on a surrogate inherits the surrogate's assumptions as blind spots, and those blind spots are invisible from inside the apparatus.

The generalised question, which now runs against every guard we write: ask not only can it fire, but is the failure it targets representable in the model it runs against?

The repair has its own trap

The natural way to add a state-dependent variance term is to zero-initialise its output projection so the module “starts as a no-op”. That reproduces the defect exactly. The quantity is constant at step 0, a firing test passes while measuring nothing, and heteroscedasticity appears only if training happens to find it — a shape, not a mechanism.

Both repairs are therefore initialised non-zero and from the physics, and the test asserts the un-repaired arm's spread is exactly 0.0 rather than describing it as constant. Asserting the dead case is what makes the live case evidence.

Two further rulings came out of it. The instrument noise floor is separately parameterised from the state term — electrode impedance genuinely is not a function of neural state, and keeping them distinct stops the floor silently absorbing the structure. And horizon dependence must come from integrating the state, never from passing the horizon index to the head: a variance that grows because it was handed h would vary with horizon for reasons unrelated to the structured state the ablation exists to measure.33This is the decorative-guard failure reproduced inside the repair, which is why it is a standing ruling rather than a code review comment. The residual horizon embedding was dropped outright: 2.1% of the gap is not worth confounding the ablation for.

The baseline that was not the baseline

One more finding from the same audit, because it has the same shape. The “subject-specific autoregressive” baseline reported 77,248 parameters. The split is participant-disjoint — verified, 71/11/27 — so every test window misses the per-subject model store and falls through to a shared 4,160-parameter fallback.

The 77,248 parameters are 71 per-subject models that are never used. The 4,160 that are used go unreported. The internal counter records only fit-time fallbacks, so the descriptor reports 71 models in use when the true count is zero — a guard watching the wrong door, in a class whose own docstring names that hazard.

What was actually wrong, in one table

Four defects, each of which looked like a separate finding. They are one class: a thing that is not the hypothesis, differing between arms, at a place nobody was looking because it is not “the model”.

Capacity budgets cover the model and the optimiser. Not one of these four is a budget. A matched budget with an unmatched interface, head, score or split is an unmatched comparison wearing a green check — and that is now a standing ruling: between-arm parity is checked along the whole path from state to scalar, not only in the budgets.

Next: checks that cannot fail →