Engineering · 08
The simpler arm is never the one that ships
Seven defects stood between a configured run and its first prediction. Six of them were the same defect wearing six different files, and none of them was a guard failing — it was a test-coverage geometry failing, repeatedly and invisibly.
The shape
The model has two arms. The control arm is one operator over a uniform state. The family arm partitions regions into families, each with its own backend and its own slice of the anatomical prior. The control arm has no mechanistic families and no family layout, so it silently skips every branch that exists to serve the other one.
The test suite ran on the control arm, on CPU. The thing that ships is the family arm, on a GPU. Both axes drifted, and six defects lived in the gap.
| site | exercised on |
|---|---|
| trainer, three rollout sites | control arm has no mechanistic families |
zero_pad multiplied by a CPU mask | tests run on CPU |
index/gather/scatter returned CPU tensors | tests run on CPU |
residual_penalty read the wrong attribute | control arm only |
| the serving path never bound its parameters | built against the previous run |
| the intervention path, twice | same |
Why two of them were worse than a crash
Most of these raised an exception, which is the lucky outcome. Two would not have.
The trainer defect: without binding the conditioning parameters, seven engineered subcortical backends run on their defaults. The anatomical conditioning that is the entire point of per-family backends is discarded, and the run completes. Loss falls. Nothing downstream can tell.
The regulariser defect: residual_penalty read an attribute that
is None on the family arm. It crashed — but had it resolved to
anything benign, the term that stops the learned residual from replacing the
mechanistic one would have been absent from exactly the arm whose
mechanistic backends it exists to protect. The learned operator would have
been free to absorb the dynamics, and the ablation would have measured a
residual block wearing a mechanistic model's name.
These do not produce a wrong number. They produce a right-looking number from a model that quietly lost the mechanism the experiment exists to test.
The rule
A test that never runs the shipping configuration on the shipping device is not testing the shipping code.
Two axes have to be crossed, not one. A suite that is green on the simple configuration and on CPU says nothing whatsoever about a run that ships the complex configuration on a GPU.
The repair, and what makes it non-vacuous
Seven tests that build the family arm and exercise exactly these branches. Two details matter more than the coverage:
- One test asserts the arm actually has mechanistic families. Without it, every other test in the file would pass vacuously the day someone changes the partition.
- One deliberately clears the bound parameters and requires the guard to raise. A guard nobody has watched fire is a guard nobody knows works.
And the rollout is exercised forward and backward, because the original crash was in a rollout and a build-only smoke test would have sailed straight past it.
Why it is a class, not an incident
Three entries already in our register have this shape: a synthetic-anatomy
fallback where the exercised path and the shipped path diverged; a
None default that hid a partition bug; a safety-relevant default
that four tests guarded and none ever let choose.
So it is filed as a class rather than a fourth instance:
Wherever a codebase has two arms, the tests will drift onto the simpler one — and the simpler one is never the one that ships.