Research / Evaluation and verification

What Survives Control Calibration? A Full-Scope Negative Result for a Locked Minimum-Description Acceptance Criterion

Stress-tests a locked control-calibrated minimum-description acceptance criterion for causal abstractions across three settings and distills three design lessons for future mechanistic-interpretability evidence standards.

Published
3 April 2026
Author
Ali Uyar
Code
github.com/aliuyar1234/mdca-build-pack

Abstract

Mechanistic interpretability still lacks clear acceptance criteria for when a neural network should count as implementing a proposed high-level algorithm rather than merely admitting a good post-hoc fit. We evaluate a deliberately rigid criterion that scores candidate causal abstractions by structural description length plus held-out residual code length and accepts them only if they beat matched spurious controls at comparable complexity, clear a grouped-bootstrap test-gap bound, and retain a positive shift gap without refitting. The paper uses only the final full-scope reruns that restore the entire locked candidate pool in three settings: a planted symbolic generator (S1), a miniature learned IOI transformer (S2), and GPT-2-small IOI (S3). No supported abstraction class is certified under either the primary or quantized robustness codebook. The negative result is nevertheless informative rather than empty. Null calibration changes decisions in all three settings; the exact planted oracle abstraction in S1 is frontier-defined yet still yields g_test = 0 and g_shift = 0; the closest frontier-defined candidates in S2 and S3 remain negative on both test and shift; and S3 retains many logged unevaluable high-complexity cells. We extract three design lessons for future evidence standards: control calibration is necessary but insufficient, support criteria must report frontier-domain exclusions separately from null-gap failure, and full configured-versus-realized candidate-pool accounting is part of the scientific claim. The result is a full-scope negative finding about this locked criterion family, not a universal impossibility theorem for mechanistic interpretability.

Why it matters

Mechanistic interpretability lacks clear criteria for when a network implements a proposed algorithm rather than merely admitting a good post-hoc fit. Running one locked criterion over the full candidate pool in three settings yields three lessons: control calibration is necessary but not sufficient, frontier-domain exclusions should be reported apart from null-gap failures, and full candidate-pool accounting is part of the claim.

Cite

@misc{uyar2026what,
  title     = {What Survives Control Calibration? A Full-Scope Negative Result for a Locked Minimum-Description Acceptance Criterion},
  author    = {Uyar, Ali},
  year      = {2026},
  url       = {https://aliuyar.dev/research/what-survives-control-calibration/}
}
AU-15
Candidate residual block: held-out residual bits L_resFrontier residual block: 5th-percentile residual of matched controls in the same 2-bit structural binCase: 4 high-level models × 3 site budgets × 10 map cells = 120 slots; tray: valid bins, test row and shift rowEmpty slot: cell logged as unevaluable; empty tray pocket: one fewer valid binThe paper fixes an acceptance rule for causal abstractions in advance and reruns its full configured pool of 120 cells in a planted symbolic generator (S1), a two-layer IOI transformer trained from scratch (S2) and GPT-2-small on IOI (S3), accounting for every cell as recorded or logged unevaluable. Each candidate is scored as structural bits plus held-out residual bits L_res and gauged against a frontier built from matched spurious controls, their 5th-percentile residual in the candidate's 2-bit structural bin, on the test split and again on a shift split without refitting. Calibrating against these controls changes the decision in every setting, and support needs three gates at once: within 0.01 bits/example of the best candidate, a grouped-bootstrap lower bound on g_test = frontier − L_res above zero, and g_shift above zero. The readout's Δ is how far, in bits/example, the closest frontier-defined candidate sits from the best one, and the gauge reads that candidate's g; in S1 that candidate and the planted oracle A★ each tie the frontier at g = 0 on both splits.