The failures that were not science

Research codebases fail at their seams. Every defect below sits at a boundary between two systems that each behaved correctly — git and symlinks, exception handling and fallbacks, a provenance hash and a log file it writes to.

A trailing slash that could not match a symlink

Large assets in this repository are git-ignored symlinks into a shared data store. The ignore file said:

assets/
data/

A trailing slash matches a directory only. It does not match a symlink of the same name. So the ignore rule that everyone believed covered assets left the assets symlink perfectly trackable, and a single git add -A committed it.

What got committed was a blob containing the symlink's target string — and that target was a worktree-local path. Checking it out in the main repository therefore made assets point at itself. Every worktree resolving through it got ELOOP: too many levels of symbolic links.

The fix is one line, and it is in the ignore file with the explanation attached, because the next person will otherwise delete the apparently redundant half:

# Both forms of each name are required. A trailing slash matches a DIRECTORY
# ONLY -- it does not match a symlink of the same name.
assets
assets/
data
data/

The sequel, which is worse because it does not look like a bug

Those symlinks are git-ignored and untracked. Merging the main branch into a worktree deletes them — git removes the paths that the untracking commits dropped, and nothing puts them back.

The failure presents as a missing dataset, not as a broken link. So the next person to hit it goes looking for the data, not for the symlink. It misdiagnosed three contributors before anyone wrote it down.

The remedy is a mechanism rather than a note in a wiki: a small idempotent script that recreates the links and then proves the result resolves rather than trusting that the link command succeeded. It also handles the failure mode that looks identical — a stale real directory left behind by a test run that regenerated the assets tree for real.

An exception swallowed by a bare except

This is the most expensive one on the page, because it made a whole subsystem silently optional.

The adapter between the anatomy prior and the foundation model raised on an interface mismatch. The exception was caught by a bare except Exception, and every run fell back — silently — to a labelled synthetic connectome.

The consequences compound in an unpleasant direction. No receptor data reached the corpus, the training run, or any checkpoint variant, because the anatomy module never loaded at all. Every provenance record was correct: the fallback was honestly labelled as synthetic. Nothing lied. It simply meant that the anatomy prior — the thing several essays on this site describe building carefully — was not in the artifact.

Two things make this hard to catch and both are general:

• The fallback worked. Training ran, losses descended, checkpoints were emitted. A silent fallback that produces a working system is indistinguishable from success at every level except the one nobody was checking.
• The label was honest. The synthetic prior was marked synthetic throughout. The defect was not mislabelling — it was that nobody had asked why the label kept saying synthetic on runs that were supposed to use the real prior.

A bare except Exception around a subsystem's construction converts “this component is broken” into “this component is optional”. If a component is genuinely optional, the fallback should be declared and asserted at the call site, not implied by a swallowed exception.

A provenance field that could never read clean

Artifacts are stamped with a git hash carrying a -dirty suffix when the working tree has uncommitted changes. Good practice, and the mechanism worked exactly as designed.

The training run writes its logs to tracked files. So by the time any checkpoint is stamped, the tree is always dirty. Every checkpoint this project has ever produced is stamped -dirty.

It was found by asking a question that costs nothing: what would this field look like if it were working? The answer — “sometimes clean” — had never been observed, and no one had noticed that they had never observed it.

The sharpest part is where it sits. This field is inside the mechanism built to catch stale artifacts, and it was about to be handed to a new provenance-enforcement gate that would have consumed a signal structurally incapable of ever reading clean. A guard built on a decorative input inherits the decoration.

A process check that matched itself

Three instances in one night, to two different people.

pgrep -f <pattern> matches its own command line. The shell wrapper running the check contains the pattern, so the guard reliably reports finding the process it is checking for.

Instance 2 was written into a brief for another person after instance 1 was known. Instance 3 was made by the same party that wrote instance 2.

Knowing a failure, and having documented it for others, did not prevent making it again.

Which is the general lesson stated as strongly as it can be, and it argues for mechanisms over knowledge: match on the interpreter and the module together and exclude the shell, or write the pattern so that it cannot describe the checking process.

Verifying through a path production does not take

Three more, same shape: the check exercised a path the production code does not take, so it passed while production failed.

A verification is only evidence about the path it exercises. The remedy is mechanical: run the check through the exact call production makes, not an equivalent one. Where that is impractical, verify the observable consequence in production output — which is why a line in the training log matters more than a passing test.

An evidentiary claim that only held where it was written

Someone verified that training source was unmodified during a run, and recorded the command:

git diff 7f18528 HEAD -- scwbd configs

HEAD is a moving, worktree-local symbol. The claim was true in the worktree where it was written and false in the next person's, where the base commit is not an ancestor of the main branch — so the diff is dominated by unmerged work on both sides and reports the opposite conclusion.

A claim that only holds where it was written is not evidence. It is a memory.

Three rules follow, all learned here the hard way. No moving symbols in an evidentiary claim — not HEAD, not a branch name, not “current”; two immutable hashes or it is not checkable. Name the branch or worktree when the commits are not on the shared trunk. And assume the reader is elsewhere and later, which is the entire point of handing someone a verification command.

Seven trainable tensors outside the permission system

The last one, included because it was caught by a guard doing exactly its job — and because it shows what a working version of everything above looks like.

A test exists specifically to catch a declared gradient permission that governs no tensor. After a merge it started failing, and it was right to: a new typed observation boundary had introduced seven trainable tensors with no entry in either permission table. They were outside the gradient permission system while it continued to look enforced.

That is the exact defect the module's own docstring was written about, caught by the module's own test, and traced with git log -S rather than guessed at. This is what the rest of this essay is arguing for.

← Back to the engineering essays