Engineering · 10
An instrument that fails silently is worse than one that fails
Our register catalogues guards that watched the wrong thing. This is different: a check that watched nothing at all, and reported it the same way it would have reported success. It was the only thing watching a multi-hour unattended training run.
The command
Checking on a training run mid-flight:
tail -3 logs/run2_pilot.log ; echo "procs: $(pgrep -cf 'scwbd.foundation.train')"logs/run2_pilot.log does not exist. The real log lives at
reports/training/run002.log.
tail on a missing path inside a compound command printed nothing
and did not stop the chain. pgrep then printed a healthy process
count. No error, no stage line, processes alive.
That output is shape-identical to a healthy run observed between log writes.
Why this is worse than a decorative guard
A guard that watches the wrong thing fails when something goes wrong — badly, but at a moment when there is a symptom to find. A silent instrument fails continuously and invisibly, and its failure is indistinguishable from the good case by construction.
Nothing was going wrong at that moment. So there was no symptom whose absence could be noticed.
An instrument whose failure output is a subset of its success output cannot be trusted, because there is no observation that distinguishes them.
Absence of an error is not evidence. tail reports a missing file
by printing nothing — and nothing is exactly what a quiet log looks
like.
What was actually exposed
A ten-minute cron watchdog was the only thing watching a multi-hour unattended run whose end state is a published model. A watchdog reading a wrong path reports healthy every ten minutes indefinitely, and the run's death surfaces only when somebody asks for the finished artifact.
The cron was, in fact, reading the correct path — its output carried real stage lines. The typo was in an ad-hoc check. That is luck rather than design: both callers were re-deriving the same path by hand on every cycle, and one of them got it wrong.
The fix
One health check, and the ordering is the entire point: the instrument is validated before the subject.
| condition | exit | what it means |
|---|---|---|
| log missing, unreadable, empty, or stale | 1 | you do not know anything yet |
global_step unparseable | 1 | wrong log, or the format changed |
| no process, or a traceback | 2 | the job is dead — relaunch |
| progressing | 0 | healthy |
Broken-instrument and dead-job are separate exit codes because they demand opposite responses. One means relaunch training. The other means do not act on anything you just read.
Staleness counts as a precondition, not a finding: a log nobody is writing to is a broken instrument, not a quiet run.
Its control
All three failure paths were run and watched to fire — including the original typo, verbatim:
UNHEALTHY(1): log not found: logs/run2_pilot.log (checked from /home/.../scales-and-dynamics)
UNHEALTHY(1): log stale: no write for 61s (limit 1s)
HEALTHY stage=T3_population_prior global_step=3846 nll=0.5045 npe_rejected=0 procs=6 log_age=61sThis is the discipline the earlier entries in the register earned — a guard nobody has watched fire is a guard nobody knows works — applied to a checker rather than to a refusal. The failure message names both the path it searched for and the directory it searched from, because the whole defect was looking in the wrong place with confidence.
The rule we carry
A monitoring command must fail loud on its own preconditions before reporting on its subject. Where a path has more than one caller — a human and a cron — derive it once. Two hand-derivations of the same path are one typo away from this page.