Engineering
What went wrong, and what it was worth
Six essays on defects found while building SC-WBD. Most were found by instruments we had built to find defects; several were found inside those instruments.
These are written for engineers and researchers, and they assume you would rather read the failure than the summary. Nothing here is retrospective tidying — every claim is traceable to a committed file, including the ones that make us look bad.
-
01
We built the control arm of our own ablation and shipped it under the model's name
Our paper's first required comparison is structured regional state versus a pooled vector. We built the second one, named it after the first, and measured it losing. The controlling defect was not the narrowing — it was that nobody had written the narrowing down.
-
02
The most expensive 134 seconds
A parameter with a closed-form optimum was left to gradient descent for two minutes, got 20% of the way there, and lost the entire run. The model had the best conditional mean of all seven arms and the second-worst score.
-
03
Decorative guards: checks that cannot fail
A catalogue of instruments whose reading was constant with respect to the question being asked. All were green. All looked like evidence. Plus the inverse category — guards asserting a defect still exists, which fail the moment someone fixes it.
-
04
A measured FAIL, kept as the deliverable
We built the fine/coarse resolution pair the paper asks for, measured whether it validates at its boundary, and it does not. The failing measurement is the result — and it says something specific about what regional state can carry.
-
05
Declaring where the implementation is narrower than the paper
Every divergence between what we wrote and what we built, in a register any contributor may add to and none may delete from. A stated narrowing is a decision you can attack. An unstated one is invisible to a process built entirely out of attacking stated things.
-
06
The failures that were not science
A trailing slash in
.gitignorethat could not match a symlink and gave seven checkoutsELOOP. An anatomy loader whose exception was swallowed by a bareexcept. A provenance field that was structurally incapable of ever reading clean. -
07
Three ways to break a density
A normalizing flow reported −log q of 4.489e9, bit-identical across runs, from the first step. Three separate causes, each invisible to the others' tests. What found them was a counter, not a bound — and the fourth fix was committed as "the root cause" and then withdrawn.
-
08
The simpler arm is never the one that ships
Six of seven defects blocking a training run were the same defect wearing six different files. Two of them would not have crashed — they would have produced a right-looking number from a model that quietly lost the mechanism the experiment exists to test.
-
09
A guard on one verb is not a guard on a noun
We built a refusal to stop a model shipping under a name it had not earned. Then an evaluation stamped the previous model's name on its own results, in the file the public card reads, and nothing objected — the guard watches emission, not writing.
-
10
An instrument that fails silently is worse than one that fails
A health check read a log path that did not exist. It printed nothing and exited zero — output shape-identical to a healthy run — and it was the only thing watching a multi-hour unattended job.
-
11
Two montages are two views of one carrier
Relating two supports meant hand-writing a map. An algebra now derives what the declarations support and refuses the rest — because a fabricated correspondence is indistinguishable at the type level from a real one. Nine sources, one view, and the lead field is a buffer.
-
12
Nine hours of training on the wrong data, with every dashboard green
We renamed the training stages. Six gates matched on the old names and five gave the wrong answer, so the model trained on simulation alone. A complete fix had been sitting unapplied since that morning, over six red tests that named the defect.
-
13
Everything needed to admit fMRI already existed
A 513-line registration module, the volumetric atlas it needs, the template it registers to, and the solver — all present, all unused, and every source card describing the capability as missing. The distance to a second modality was a caller, not a component.
The one heuristic worth taking away
If you read nothing else here, take this. It found two of the most expensive defects in the list above, before either instrument had ever been run, and it costs nothing:
For every quantity your code names as varying, ask what it is a function of — and read the answer off the expression rather than the docstring. If the expression's free variables do not include the thing it claims to depend on, it is decorative. No run required, no data required.
The companion question, for guards specifically:
Take the failure this guard exists to catch. Apply it to the guard's input. Does the reading move? If not, the guard is decorative — however sound its mechanism, however green its tests, and especially if it has never been run.
And the corollary that is harder to see, because from inside the apparatus every check is green and every check is correct:
Ask of any guard not only “can it fire” but “is the failure it targets representable in the model it runs against”. A verification apparatus built on a surrogate inherits the surrogate's assumptions as blind spots.