Decorative guards: checks that cannot fail

Every instance below was green, plausible and load-bearing. Because a decorative guard looks like evidence, it is worse than no instrument at all — confident reasoning follows from it. Seven of them were found inside the machinery built to catch exactly this.

The tell

In every case the instrument returned the same value under both branches of the question it was asked.

Four distinct variants, because the fixes differ

1. Absence — where a missing record looks exactly like a clean one

Two of the sharpest cases share this form. No -dirty suffix would mean clean source; no clamp record would mean the prior fit inside the backend's support. In both cases the absence was produced by the mechanism failing to run at all — and absence is exactly what success looks like.

The provenance case is the one that stings: the run writes to tracked files as it goes, so the artifact is always dirty. Every checkpoint this project has ever produced is stamped -dirty, and the field was about to be handed to a new provenance-enforcement gate that would have consumed a signal structurally incapable of ever reading clean.

The clamp case: a helper resolved a lookup key to None whenever a backend spelled its timescale parameter differently, then skipped the block silently, writing no provenance entry. A perfect 0% clamp rate meant the prior had never arrived at all. It affected 21.6% of the corpus, and it was found by asking why two backends had scored a suspiciously perfect zero.

A field that is only ever written on success is not a record. It is a wish.

This variant cannot be caught by looking harder at the output, because there is no output to look at. The remedy is to make the null case write something — when a mapping is undisclosed, record {"disclosed": false} explicitly, so that silence never reads as safe.

2. The wrong question, asked of a fine instrument

The composite training loss is a perfectly good instrument. It is a weighted sum whose terms move for unrelated reasons, so comparing two runs on it conflates “the model forecasts better” with “a different term happens to dominate right now”.

It misled twice in one hour, in opposite directions. At step 80 it showed 3.717 against 2.447 — a large apparent effect — where the interpretable forecast metric read 20.943 against 20.980, no effect at all. Over steps 40–140 it showed a consistent advantage that reversed by step 160, while forecast quality tracked within 2% throughout.

For a comparison, choose the narrowest metric that answers the actual question — and choose it before seeing which one is favourable. An aggregate is for monitoring, not for adjudicating.

3. The preserved quantity — the cleanest case here

This one was caught before the instrument had ever run, and it is the most transferable.

The paper warns that a lower-variance model is not preferred when it achieves stability by smoothing away the effect of interest. We built a check that executes that warning, and built it well: it applies an effect function identically to the truth and to both predictions, so “the model is smoother” and “the model lost the effect” are separated rather than conflated, and it bootstraps the retention ratio. The mechanism is sound.

All of its discriminating power lives in one callable, and the default was global dynamic range. Now write down the failure mode it exists to catch: a heterogeneous model fails by collapsing every region onto one shared dynamic. Take any signal and rescale each channel to the pooled temporal standard deviation:

Between-region differentiation is destroyed. Global dynamic range is unchanged — not approximately, exactly.

The check would have run, returned a retention ratio near 1.0, reported “not smoothing”, and been structurally incapable of any other reading on the one input it exists to catch. It would then have sat in the report as evidence that the model's win was earned.

The instrument is right; the question is right; the conserved quantity is wrong. That is invisible to code review, invisible to tests that assert the check runs, and invisible to a passing negative control — unless the negative control is built out of the failure mode itself. The replacement test asserts both halves, because only the pair is informative: the old effect function is blind to the collapse, and the new one sees it.

4. Unrepresentable — the guard is sound and the surrogate cannot express the failure

Covered in essay 02. The identifiability checks run on a linear-Gaussian surrogate in which state-independent innovation covariance is a theorem, so a variance parameter with no path from state was not merely undetected — it was unrepresentable. Every check was green and every check was correct.

The inverse category: guards that assert a defect still exists

The mirror image, and it is easy to miss because these guards fail loudly — which makes them recoverable, but also makes them look like someone else's bug.

One test asserted that a fallback connectome is labelled not biological. Sound, and it fired correctly for months. Then the loader was hardened so that it refuses to substitute the synthetic prior silently — it now raises unless explicitly forced. The test's fixture does not force it, so the path the guard polices is never entered. Where the assets are absent it raises; where they are present it would load the real prior and the assertion would fail.

The general form is a test that encodes “this is still broken”. It is a legitimate thing to write, and it must be labelled as such, because otherwise the person who fixes the defect is greeted by a red suite and no explanation of why fixing something broke the build.

A preregistered threshold with no reference class

This is the headline finding, and it survived every procedural safeguard the project has.

A training-continuation trigger required a forecast metric to fall below 1.0 by step 900. It was written before the data existed, never moved, honoured to the letter when it resolved, escalated the moment a sibling clause fired, and adjudicated by a party who was not its author. Every safeguard worked, in order, correctly.

The result — not met, 1.1200 — is still uninterpretable. To know whether “below 1.0” was a reasonable ask of a 1.76M-parameter model, on 37 simulated shards, with roughly 40% of its regional-timescale prior missing or clamped, you need a reference class: a capacity-matched baseline, a matched control, the same architecture at another budget. The project had none. Without one, the threshold cannot distinguish a model that underperformed from a number that was never achievable. Both produce an identical reading.

A preregistered threshold with no reference class is a guess with a timestamp.

Note what this is not. It is not “be more disciplined” — discipline was not the binding constraint and more of it would not have helped. Process discipline cannot manufacture a reference class. The bar was inappropriate, and not because it was too high; height was never the issue, it had no scale.

The fix: state a threshold's reference class when you set it, or set a matched control instead of an absolute value. An absolute number encodes an assumption about what is achievable, and that assumption becomes invisible the moment the number is written down. A control encodes no such assumption. Prefer controls — under an instrument rescale, both sides of a comparison move together and the comparison survives.

And what a preregistration inherits

The same threshold demonstrates a limitation of preregistration as a technique. Midway through, a normaliser defect was found and fixed, and the metric's scale changed by two orders of magnitude:

The restart fixed the signal the bar was evaluated on. It did not fix the bar — because the bar had been chosen by looking at the same corrupted numbers. Half the contamination was repaired; the other half was invisible precisely because it had already been written down. A committed number stops attracting scrutiny, which is most of its value and, here, exactly the problem.

When the instrument a preregistration was written against is found defective, the preregistration does not become wrong. It becomes uninterpretable, and the honest response is to report it as uninterpretable rather than to re-set it — regardless of which direction the new bar would move. A harder bar is not a cleaner one. Report it in three layers that are never merged: the literal fact, the interpretation, and the adjudication — with the third decided by someone who is not the author. Merging the first two lets a caveat quietly do the work of a result; merging the last two lets the author grade their own homework.

The human variants

The rows above are mechanisms that cannot discriminate. These are processes that cannot discriminate, and they are the ones the project actually ran into.

A conclusion nobody is attacking

A conclusion that survives only because everyone has become invested in it reads exactly like one that survives because it is true. Same confidence, same articulacy, same absence of dissent. Nobody lied; the mechanism that would have produced disagreement quietly stopped running — the absence variant again, in people.

Both directions appeared within one hour on the same decision: the author reaching for the reading that makes their most recent action look correct, and the reviewer under-auditing evidence because it arrived well-argued. The second is more dangerous, because it converts one party's error into everyone's, and it gets stronger the better the collaboration is going.

A conclusion nobody is trying to break is not a finding. It is a consensus.

True, and not the proposition at issue

The most durable of all of these, because attacking it confirms it.

A rule said: commit nothing until the artifact freezes its identity. Three commits went in anyway, each justified by “docs-only commits cannot change the model.” That justification is correct. It is simply not the proposition the rule was protecting. There was nothing false to notice, no contradiction to trip over — and the substituted rule was strictly easier to satisfy.

A rationalisation that is false gets caught; one that is true but irrelevant does not. The check is not “is this reasoning sound?” — it will be. The check is: which proposition does this argument establish, and is it the one at issue?

Auditing the incentive instead of the argument

An argument that disadvantages its own author feels checked on arrival. It is not. One such argument was accepted for exactly that reason, called sharp, and turned out to be wrong on inspection.

Accepting a generous argument from the party it disadvantages is the same error as accepting a flattering one from the party it favours. Both substitute a judgement about incentive for a judgement about validity. The first feels like rigour, which makes it the more durable of the two.

A true explanation that does not exhaust the cause

A training stage replayed after a resume. The diagnosis — resume granularity is stage-level, so an interrupted stage replays from step 1 — was true, and it is a real design property. It was also incomplete: a completed stage was replaying too, because the bookkeeping that records a stage as finished ran after the checkpoint write, so no stage ever recorded itself as done.

A wrong explanation gets falsified by the next observation. A right but partial one absorbs it. The correctness is load-bearing for the error. The operative check: does this explanation account for the full magnitude of what I observed, or only for its existence?

A descriptive range that becomes a threshold by drift

A quantity was explicitly ruled a diagnostic, not a criterion. Someone then observed it oscillating and wrote “I will not report this again unless it leaves the 5–14 band.” The band was descriptive — a range that had been seen. Attaching a reporting decision to it made it a criterion. Nobody decided to set a threshold; it assembled itself out of an observation plus a commitment, and later fired on a sample 0.6% over an edge chosen by eye.

A descriptive range with a decision attached to it is a threshold, however it was arrived at. Report the trajectory, not the band.

The standard we now hold, and it is stricter than the one we recommended

This document spent a long time saying: write a negative control; show the guard can fire. That is the weaker form. It constructs a failing input and asserts the guard goes red.

The stronger form mutates the implementation. Stub the validator to a no-op and confirm all fourteen guards fail. Replace the definition they protect with a structurally similar but meaningless one, and confirm the test still fails — against a matched null control, so that “a regrouping fails” cannot just mean “any regrouping fails”.

A firing test proves the guard responds to a bad input you thought of. A mutation test proves the guard is load-bearing on the code it protects.

Several of our own firing tests are the weaker form and are recorded as not yet at this standard, rather than described as if they were.

Three habits, in the order they are worth

1. Make the guard fail on purpose before trusting it. Break the thing deliberately and confirm the alarm sounds. Doing this to a memory cap caught a real bug in the fix itself — a guard that raised an error instead of enforcing a limit, which is the same failure one level up.
2. Say what you expect to see if the thing is broken, before reading. If that is the same as what you expect if it is fine, stop and find a different instrument.
3. Ask of every caveat: if this were true, would the claim change? If not, the caveat is ornament and the claim is unearned. A retracted periodicity claim here carried an accurate, freely-volunteered aliasing caveat — and then asserted the period anyway. The caveat was correct, prominent, and load-bearing on nothing, and it bought unearned credibility precisely because it signalled awareness of the limitation it failed to apply.

And its sibling, which is worse: a true qualifier that increases apparent confidence while the underlying measurement stays weak. We reported “2.24 s/step, settled, warm-up excluded”. Every word true — excluding warm-up is correct practice. But a reader hears the confound has been handled, when it actually meant this is a 60-step sample taken immediately after warm-up, precisely where a rate is least likely to have converged. The settled figure was 2.95, thirty-two percent higher. Warm-up was nameable; sample length was the larger error and went unnamed.

Next: a measured FAIL, kept as the deliverable →