ScrubID: Identifiability-Aware Auditing for Mechanistic Interpretability

Mechanistic interpretability often claims that a specific circuit explains a model behavior, but multiple distinct circuits can reproduce the same behavior under common intervention protocols. We introduce ScrubID, an identifiability-aware auditing framework that treats circuit explanations as estimators whose uniqueness depends on the intervention family. ScrubID defines scrubbed models that preserve a candidate circuit while resampling the rest of the network under a specified intervention family, and evaluates ε-faithfulness relative to a task-specific tolerance. To detect non-identifiability in practice, ScrubID introduces three diagnostics computed from discovered candidate sets: RR, a redundancy ratio measuring worst-case disagreement among near-optimal faithful circuits; SSS, a stability score measuring how consistent recovered circuits are across replicate discovery runs; and CC, a contradiction score measuring how inconsistent component necessity is across near-optimal faithful circuits. We validate these diagnostics on a deterministic synthetic suite with ground-truth equivalence class sizes and planted redundancy, and we apply ScrubID to real transformer case studies on indirect object identification, year-span greater-than comparison (yes/no), and induction-style next-token prediction. Our results motivate reporting identifiability diagnostics alongside mechanistic circuit claims and provide standardized artifacts for reproducible audits.

On the synthetic suite, RR and CC jump from 0 at planted redundancy factor 1 to RR ≈ 0.67 and CC ≈ 0.67–0.82 for redundancy factors ≥ 2, and audit certificates are emitted at rate 1.0 in these redundant regimes (Table T1). In GPT-2 case studies under activation patching, across N=5 suite seeds we compare three candidate generators (G_ATTR_PATCH, G_MANUAL_SEED, G_RANDOM_K). On the ID split, G_ATTR_PATCH recovers smaller ε-faithful circuits for IOI (mean 124.0/156) and induction (mean 92.8/156); IOI exhibits moderate redundancy/contradiction signals (RR_mean ≈ 0.36, CC_mean ≈ 0.10) while induction shows both redundancy and discovery instability (RR_mean ≈ 0.59, SSS_mean ≈ 0.41), producing WARN/FAIL verdicts (Table T2). On the OOD split, induction becomes highly unstable (SSS_mean ≈ 0.13) and exhibits RR non-identifiability in some seeds (e.g. RR=0.93 with |S_near|=2), producing audit certificates (Table T4).