When Accuracy Hides Path Dependence: Criterion Fidelity in LLM Judges

Large language model judges are commonly validated by ordinary task accuracy or judge–reference agreement, implicitly assuming that a judge that looks accurate is following the provided criterion. We argue that this is incomplete: in criterion-conditioned evaluation, a valid judge should change when the criterion meaning changes and remain stable when wording changes but meaning does not. We formalize this property as criterion fidelity and measure it with criterion families—matched groups of base, paraphrase, and counterfactual variants evaluated with mirrored-order continuation scoring. Across two controlled task families—QA-Key, a high-margin keyed-answer setting, and BioRubric, a low-margin rubric-conditioned pairwise setting where both candidates are factually true—ordinary quality and criterion fidelity dissociate. Under standard prompting, QA-Key gaps are small but real, while BioRubric gaps are large: for Qwen2.5-14B-Instruct, BaseAcc = 0.783 but GCF = 0.498; for Llama-3.1-8B-Instruct, BaseAcc = 0.727 but GCF = 0.217. Strict criterion-emphasis prompting is not a reliable remedy: it removes the gap on QA-Key/Qwen2.5-14B-Instruct, narrows it on BioRubric/Qwen2.5-14B-Instruct, and worsens both quality and order instability on BioRubric/Llama-3.1-8B-Instruct. Failure analysis shows that underreaction to criterion changes dominates paraphrase instability, but the largest rubric-based failures are often mediated by presentation-sensitive path dependence rather than by a clean majority of direct semantic criterion refusal. On the most pathological slice, Llama-3.1-8B-Instruct evaluated with the strict prompt becomes nearly a pure first-position chooser on BioRubric while remaining almost balanced over underlying candidate identity. These results position criterion fidelity as a distinct validity axis for LLM judges and show that criterion-conditioned instability, not just answer-level inaccuracy, should be part of judge evaluation.