The identifiability benchmark returned a verdict
Three regions, linear-Gaussian, five designs, a decision rule written before the run. C1 and C2 are the thesis's first differentiator, and as measured they do not hold.
Applies to no checkpoint — a method benchmark on a three-region linear-Gaussian system with ground truth. The verdict is about the joint-inference claim itself, so it stands until that claim is re-tested.
SC-WBD's first claim is that fusing EEG and fMRI at their native sampling rates identifies coupling and delay parameters that neither modality resolves alone, and that resampling to a common clock destroys the advantage. That is testable on a system small enough to have ground truth, so it was tested on one: three regions, linear dynamics, Gaussian noise, a 12 ms conduction delay and an explicit haemodynamic lag.
The rule, written first
Five criteria, each quantified over the minimum eigenvalue of the profile information matrix for the pre-registered parameters, evaluated in every regime.
| Criterion | Rule |
|---|---|
C1_fusion_information | joint-native ≥ 1.05 × max(EEG-only, fMRI-only), in every regime |
C2_native_beats_resampled | joint-native > the naive-resampling estimator |
C3_intervention_information | impulse-matched ≥ 1.05 × joint-native |
C4_calibrated_recovery | nominal 95% inside the Wilson interval of empirical coverage |
C5_recovery_improvement | delay and θ RMSE ≤ the better single modality |
The result
verdict: INCOMPLETE
C1_fusion_information FAILED in every regime
C2_native_beats_resampled FAILED (passes in 1 of 3)
C3_intervention_information FAILED (passes in 1 of 3)
C4_calibrated_recovery NOT EVALUATED
C5_recovery_improvement NOT EVALUATED
| regime | C1 | C2 | C3 |
|---|---|---|---|
reference | fail | fail | fail |
low_snr_short_delay | fail | pass | pass |
weak_coupling_long_delay | fail | pass | fail |
C1 never passes. Joint native-clock inference does not reach 1.05× the better single modality in any regime tested. C2 and C3 pass where the problem is hard — low SNR, weak coupling — and fail in the reference regime, which is the one where the parameters are easiest to see.
The rule says every regime, so partial passes are failures. That was decided before the numbers existed, which is the only time it can be decided honestly.
Why two criteria are unevaluated rather than failed
The benchmark reports its own reasons, and both are properties of the experiment rather than of the model.
Convergence gating. MAP recovery ran a fixed number of Newton
steps. The preconditioner is the expected information at the prior mean
and is never refreshed, so the iteration converges linearly, not quadratically —
a fixed step count cannot certify convergence. Two regimes were gated out on
that basis. The loop now stops on the Newton decrement,
sqrt(gTH-1g), the remaining distance to the
optimum in posterior standard deviations, and records how many steps it actually
took.
A degenerate reference regime. Its truth sits exactly on the
prior mean — 0.000 prior sd on all four pre-registered parameters.
There, an estimator that ignores the data entirely and returns the prior mean scores zero bias, zero RMSE and 100% coverage. Recovery metrics cannot discriminate anything.
It is not hypothetical: fmri_only, whose information about the
delay is 5e-05, posted a delay RMSE of 0.028 ms
against EEG's 0.233 ms — eight times "better" purely by not
moving. The report now measures each regime's offset from the prior mean in prior
standard deviations and marks anything under 0.25 as unable to discriminate.
What this settles, and what it does not
It is a measured negative on the thesis's first differentiator, from a benchmark that ran to completion on a system with ground truth. It is independent of anything about the trained model — different code path, different system, no neural network involved.
It does not say fusion cannot help. It says that on this three-region linear-Gaussian system, with these five designs and this pre-registered rule, the information advantage did not appear at the 1.05× threshold the rule demanded. A larger system, a different parameterisation, or a different threshold are all live possibilities — and each would need its own rule written first.
The artifact is reports/identifiability/results.json, with the
per-design decomposition in
results_by_design_checkpoint.json and the narrative in
reports/identifiability/summary.md.