Nine hours of training on the wrong data, with every dashboard green

What was supposed to happen

Run 2 trains in six stages. Some train on simulated trajectories; some take a gradient on real EEG; the last individualises to participants. The stages carry names that say which is which — T1_measured_founding, T1_individualisation.

What happened

Run 2's config renamed every stage from the run-1 scheme. Six gates in the trainer match on the run-1 names. None was updated, and five of the six gave the wrong answer:

The fifth row is the only reason the run trained at all, and it is correct by accident — it is the one gate written as != rather than ==, and an unknown name happens to satisfy it. Nine hours of training rested on a negation nobody wrote for that purpose.

Measured against the actual config, rather than inferred:

stages with a permission entry      : NONE  -> all fall back to ("*",)
stages that compute a real-data loss: NONE
stages that build an individualizer : NONE

So the run trained for nine hours on simulated trajectories alone, with the per-stage gradient restrictions inert and no individualizer, under a stage named measured founding. The real-EEG loader was constructed and its split fingerprinted. The gradient was never taken.

The log says so at every step of every stage. The only loss field it has ever emitted is sim_forecast_nll. There is no real-data term anywhere in nine hours of output.

The shape of it

A dictionary lookup with a permissive default is a configuration system that cannot report a typo. .get(name, ("*",)) and name in (...) are unfalsifiable by construction — there is no stage name they reject.

They fail toward permissive. An unmatched name means no restriction, no real loss, no haemodynamic state and no individualizer — never an exception. Nothing crashed. Loss fell. The rejection counter held at zero. Every dashboard this run has was green for nine hours.

Our own register already had the sentence for this, written about a smaller instance: these do not produce a wrong number, they produce a right-looking number from a model that quietly lost the mechanism the experiment exists to test.

How it was found, which is the uncomfortable part

Not by a guard. Not by a dashboard. By reading the trainer to answer an unrelated question — whether a branch in the evaluation code, which had never executed in this run, would crash after the final stage. The defect was three lines above the branch being checked.

Nothing in this project would have reported it. Reading a path adjacent to the one you care about is how an untested branch gets read at all.

The worst part: the fix was already written

Everything above says the defect went undetected. It did not.

A module sits on the main branch called curriculum_admission.py. It contains an inventory of all six stage-name gates, recorded as data — in its own words, "so a test can assert the list is exhaustive rather than trusting that someone read the function carefully." It defines an exception, UndeclaredStage, whose docstring states the diagnosis better than we managed independently ten hours later:

Raised rather than defaulted. STAGE_PERMISSIONS.get(name, ("*",)) answers this question with everything, which is the one answer that cannot be wrong-looking: an unwired stage and a fully-permitted stage produce the same reading.

There is a complete 13 KB patch that rewires the trainer to take admission from the config. It still applies cleanly. And there are eleven tests whose header records the measurement in both directions — 7 failed before the patch, 11 passed after — with the three that could not discriminate marked as such, and one marked as having failed for a different reason than predicted, "worth recording rather than counting as a win."

The patch is dated 07:04 that same morning. The run started at 18:32. It was never applied. Six of those tests are red on the main branch right now, and were red for the entire nine-hour run:

FAILED test_run_stage_has_no_stage_name_gates
FAILED test_run_stage_consults_stage_admission
FAILED test_stage_sources_takes_an_admission
FAILED test_stage_sources_excludes_unadmitted_sources
FAILED test_sim_losses_takes_an_admission
FAILED test_anatomical_prior_is_not_gated_on_the_sim_batch

And the config had already said so

There is one more layer, and it is the one that changes the diagnosis.

Every stage in the run-2 config carries a block declaring exactly the properties those six gates decide by name — which data tiers the stage may use, what each tier may update, and the flags themselves. Including this comment, written before the run:

# --- the four behaviours run 1 keyed on the stage NAME -----------
# Declared, because a stage not called "I_regional" silently loses
# boundary randomisation and a stage not called "V_individual"
# silently never builds the Individualizer.
boundary_randomisation: true   # matches run 1's I_regional
with_hemo: false
individualize: false

Somebody identified this exact hazard, named it, and answered it — in the config, in advance.

The trainer never reads that block. The key does not appear anywhere in the file.

A configuration block nothing reads is decorative configuration: it has the shape of configuration, sits where configuration goes, is written with care and justified in comments — and changes nothing. It is a decorative guard on the other side of the interface, and worse in one respect: a decorative guard at least runs.

What makes it invisible is that neither half is wrong on its own terms. From the config, every behaviour is declared and correct. From the trainer, every behaviour is decided and consistent. Only reading them against each other shows the two halves never meet.

It also settles what the fix is. Not "add the new names to the tuples", not "design a declaration format" — the declaration exists and is already correct. The patch's entire job is to make the trainer read the file it was handed.

Which makes the lesson a different one

Every other entry in our defect register is about an instrument that could not see a failure. This one is about an instrument that saw it perfectly, said so in plain language, and was not read.

A red test nobody reads is worse than no test. It produces the appearance of coverage and none of the effect — and it converts "we have no check for this", which prompts action, into "the suite is a bit red", which does not.

There is nothing decorative about the name test_run_stage_has_no_stage_name_gates. It says exactly what it checks. It failed, in the same repository, for the whole run.

And the same error in miniature, ours: we had catalogued "17 pre-existing failures in one test file, known and deliberately not fixed" — and never asked what else was red. The 17 was measured. Treating it as the total was assumed.

A known-failures list that is not exhaustive is itself a permissive default: anything outside it reads as passing.

What we did, and did not do

We did not kill the run. About an hour remained, and a fix to the stage gates cannot be validated except by another full run — the same untestable-fix trap we had already recorded once this run, when a repair to the posterior could not be evaluated by any probe that failed to reproduce the original failure.

The trainer was also left untouched while the process was live. Editing it would not have affected the running job — Python does not re-read its modules — but a crash-and-resume would have continued under different rules than it began with. One run, two regimes, and no way to say which weights came from which.

What we did instead was describe the artifact accurately everywhere a reader meets it:

A simulation-trained model over a real anatomical prior, evaluated on real EEG it never saw during training. Its holdout numbers are a simulation-to-measurement transfer result, not a held-out-performance result.

That is a legitimate thing to measure. It is not what the stage names claim, and it is not what "43 GB corpus" suggests to a reader — so the model card says it above the scores, and so does this site.

The test that should have existed

Four checks: every declared stage must be known to the permission table; at least one must take a gradient on measured data; a stage named for individualisation must actually individualise; a stage named for measurement must use measurement.

All twelve cases fail today, across all three run-2 configs. They are marked xfail(strict=True) rather than skipped or deleted — so the suite stays green and does not block shipping, the failure stays runnable rather than living only in a report, and when the gates are fixed the tests XPASS, which pytest reports as a failure, forcing whoever fixed it to come back and remove the marker.

The file also says, in as many words, not to make them pass by adding the run-2 names to the tuple. That would turn the tests green without making the trainer use measured data, which is precisely the move this whole register exists to catalogue.

The fix is not a longer tuple

Adding the run-2 names to both collections works, and leaves the identical trap for run 4. The mechanisms should key on a stage property the config declares — uses_real_data, individualises — so a stage that fails to declare one is refused at config load rather than granted a silent wildcard.

All three defects are one defect: behaviour attached to a string literal that nothing checks against the config.