Research / Inside the model
Binary Context Sufficiency in Hidden States: Dissociating Sufficiency from Correctness in Fixed-Question, Changed-Context Multi-Hop QA
Shows that Qwen2.5-7B-Instruct hidden states linearly encode binary context sufficiency in multi-hop QA, separating sufficient from insufficient context at 0.952 AUROC on closed-book-negative examples, a signal that stays strong when correctness is held fixed.
- Published
- 9 March 2026
- Author
- Ali Uyar
Abstract
Context-grounded QA failures conflate at least two cases: the model may misuse evidence that is present, or the provided context may itself be insufficient. We study this distinction in fixed-question, changed-context multi-hop QA. Each HotpotQA question is converted into four controlled context variants (sufficient, sufficient-noisy, missing, misleading), and we probe hidden states at prompt end and after a short two-step reasoning prefix. The strongest supported result is binary rather than three-way. On the primary Qwen2.5-7B-Instruct run, a linear probe separates sufficient from insufficient context with 0.952 AUROC on closed-book-negative examples, and the signal remains strong when final correctness is held fixed (0.979 AUROC on correct-only and 0.939 on wrong-only closed-book-negative subsets). Title masking preserves the effect (0.936 AUROC). At the same time, prompt-end representations are at least as informative as end-of-step-2 representations, missing-vs-misleading separation remains near chance in the original three-way analysis (0.505 AUROC), and a DeepSeek-R1-Distill-Qwen-7B replication is mixed rather than confirmatory (0.803 AUROC, effectively tied with prompt end and slightly below the strongest lexical baseline). We therefore make a narrow claim: hidden states encode a coarse, correctness-dissociated context-sufficiency signal, but the current evidence does not support a richer three-way evidence-state representation or a claim that the short reasoning prefix creates the signal.
Why it matters
Context-grounded QA failures conflate evidence that is present but misused with context that is insufficient. The sufficiency signal holds on wrong answers alone (0.939 AUROC on the wrong-only closed-book-negative subset), so a probe can separate the two cases from the hidden states.
Cite
@misc{uyar2026binary,
title = {Binary Context Sufficiency in Hidden States: Dissociating Sufficiency from Correctness in Fixed-Question, Changed-Context Multi-Hop QA},
author = {Uyar, Ali},
year = {2026},
doi = {10.5281/zenodo.18921976},
publisher = {Zenodo},
url = {https://doi.org/10.5281/zenodo.18921976}
}