Research / Evaluation and verification
What Survives Control Calibration? A Full-Scope Negative Result for a Locked Minimum-Description Acceptance Criterion
Stress-tests a locked control-calibrated minimum-description acceptance criterion for causal abstractions across three settings and distills three design lessons for future mechanistic-interpretability evidence standards.
- Published
- 3 April 2026
- Author
- Ali Uyar
Abstract
Mechanistic interpretability still lacks clear acceptance criteria for when a neural network should count as implementing a proposed high-level algorithm rather than merely admitting a good post-hoc fit. We evaluate a deliberately rigid criterion that scores candidate causal abstractions by structural description length plus held-out residual code length and accepts them only if they beat matched spurious controls at comparable complexity, clear a grouped-bootstrap test-gap bound, and retain a positive shift gap without refitting. The paper uses only the final full-scope reruns that restore the entire locked candidate pool in three settings: a planted symbolic generator (S1), a miniature learned IOI transformer (S2), and GPT-2-small IOI (S3). No supported abstraction class is certified under either the primary or quantized robustness codebook. The negative result is nevertheless informative rather than empty. Null calibration changes decisions in all three settings; the exact planted oracle abstraction in S1 is frontier-defined yet still yields g_test = 0 and g_shift = 0; the closest frontier-defined candidates in S2 and S3 remain negative on both test and shift; and S3 retains many logged unevaluable high-complexity cells. We extract three design lessons for future evidence standards: control calibration is necessary but insufficient, support criteria must report frontier-domain exclusions separately from null-gap failure, and full configured-versus-realized candidate-pool accounting is part of the scientific claim. The result is a full-scope negative finding about this locked criterion family, not a universal impossibility theorem for mechanistic interpretability.
Why it matters
Mechanistic interpretability lacks clear criteria for when a network implements a proposed algorithm rather than merely admitting a good post-hoc fit. Running one locked criterion over the full candidate pool in three settings yields three lessons: control calibration is necessary but not sufficient, frontier-domain exclusions should be reported apart from null-gap failures, and full candidate-pool accounting is part of the claim.
Cite
@misc{uyar2026what,
title = {What Survives Control Calibration? A Full-Scope Negative Result for a Locked Minimum-Description Acceptance Criterion},
author = {Uyar, Ali},
year = {2026},
url = {https://aliuyar.dev/research/what-survives-control-calibration/}
}