Research / Evaluation and verification
When Formatting Flips Answers: A Deterministic Atlas of Retokenization Sensitivity in LLM Evaluation
Presents RIA, a deterministic audit isolating retokenization sensitivity in multiple-choice QA, and finds that routine formatting edits reach a 9.18% maximum flip-rate across BoolQ, ARC-Challenge and an 8-subject MMLU subset.
- Published
- 18 February 2026
- Author
- Ali Uyar
Abstract
Semantics-preserving formatting should not change model decisions, but in multiple-choice QA it often does. We present Retokenization Invariance Atlas (RIA), a deterministic audit that isolates this effect with paired no-truncation/semantics gates, scope-only token-divergence metrics, and mitigation checks with explicit no-harm criteria. Across BoolQ, ARC-Challenge, and an 8-subject MMLU subset (36,820 gated rows, two instruct models), routine formatting edits reach 9.18% maximum flip-rate (Llama3-8B, mmlu_subset, O4). TD is consistently positive but modest (ρ ∈ [0.072, 0.206], pass 1/6), while ablations expose strong template/placement sensitivity and non-universal canonicalization safety on MMLU. RIA turns retokenization sensitivity into a reproducible reliability axis with complete scripts, logs, figures, and gate evidence.
Why it matters
Formatting that preserves meaning should not change a model's decision. RIA isolates that effect with paired no-truncation and semantics gates and checks mitigations such as canonicalization against explicit no-harm criteria.
Cite
@misc{uyar2026ria,
title = {When Formatting Flips Answers: A Deterministic Atlas of Retokenization Sensitivity in LLM Evaluation},
author = {Uyar, Ali},
year = {2026},
doi = {10.5281/zenodo.18682874},
publisher = {Zenodo},
url = {https://doi.org/10.5281/zenodo.18682874}
}