ProbeRoute: Probes as Routing Priors for Frozen-Backbone Multi-Token Prediction

Frozen autoregressive language models are trained for next-token prediction, but their hidden states can still encode information about several future tokens. ProbeRoute tests whether that latent structure can be converted into a better explicit multi-token adapter without unfreezing the backbone. The method first runs a one-time offline probe stage across depth, then uses probe-derived top-5 scores to initialize a sparse top-m router over frozen hidden states. Under a stage-gated protocol with mandatory probes, screening baselines, a finalist-selection step, and a final 1B rerun, the resulting sparse adapter beats the strongest selected dense frozen-backbone baseline on the paper’s two headline held-out metrics. On the final 1B comparison, ProbeRoute improves mean_top1_h2_h4 from 0.1162 to 0.1172 and the speculative draft acceptance proxy mean_accept_len from 1.1110 to 1.1188, while changing mean_nll_h1_h4 only from 5.3769 to 5.3780. The practical point is therefore stronger than a small raw delta: the gain comes from a more selective routing interface, not from a heavier backbone adaptation recipe. The mechanistic story is equally important. Probe heatmaps reveal horizon-dependent depth structure at both 410M and 1B, the learned sparse router concentrates mass on the same depth bands, and the random-initialization ablation is weaker than probe-initialized sparse routing. By contrast, loss warmup and deeper far-horizon heads do not improve over the base sparse configuration at the tested budget. The evidence therefore supports a focused conclusion: in this frozen-backbone setting, future-token probes are not merely descriptive diagnostics; they are a useful routing prior for explicit multi-token prediction.