Research / Inside the model

Attention-Side Transformer Computations Organize Shared Multilingual Brain Alignment during Naturalistic Story Listening

Maps where shared multilingual brain alignment sits inside XLM-R-base and NLLB-200 encoders, showing it is organized primarily around attention-side and post-attention computations rather than late FFN states.

Published
23 March 2026
Author
Ali Uyar
DOI
10.5281/zenodo.19185601
Code
github.com/aliuyar1234/multilingual-transformer-neuroanatomy

Abstract

Most brain-language-model alignment studies rely on returned hidden states, leaving unclear which internal transformer computations carry cross-lingually shared signal during naturalistic comprehension. We analyzed English, French, and Chinese listeners from the Le Petit Prince multilingual naturalistic fMRI corpus (94 participants; 9 runs per language cohort) using ROI-average encoding models built from intermediate states of XLM-R-base and the encoder of NLLB-200-distilled-600M. For each sentence span and target language, we decomposed state representations into a leave-target-out shared component and an orthogonalized specific residual, then contrasted attention-side and FFN-side contributions within semantic and auditory ROI families. The planned semantic FFN-preference hypothesis failed uniformly: all six semantic model-language contrasts were negative (mean effect −0.0083; 0/6 Holm-significant in the planned direction), and supplementary reverse-direction tests favored attention-side states in all six cells. By contrast, the auditory attention-preference hypothesis was positive in 5/6 cells and Holm-significant in 4/6 (mean effect 0.0033). Representative winners concentrated in attn and post-attn rather than ffn states, semantic representative settings showed a larger mean content bias than auditory settings (0.663 vs. 0.610), and deleting top-attributed words hurt held-out prediction more than matched-random deletion in all 20 summary rows (mean top-k minus random difference 0.0381). These results do not support a late FFN-centered account of shared multilingual brain alignment. Instead, they indicate that the shared multilingual advantage is organized primarily around attention-side and post-attention computations, with the clearest confirmatory support in auditory cortex and a consistent semantic shift away from FFN-side states.

Why it matters

Brain–model alignment studies usually read returned hidden states. Reading intermediate attention-side, post-attention and FFN states instead shows which internal computations carry the signal shared across languages.

Cite

@misc{uyar2026inside,
  title     = {Attention-Side Transformer Computations Organize Shared Multilingual Brain Alignment during Naturalistic Story Listening},
  author    = {Uyar, Ali},
  year      = {2026},
  doi       = {10.5281/zenodo.19185601},
  publisher = {Zenodo},
  url       = {https://doi.org/10.5281/zenodo.19185601}
}
AU-6
Plug: the tapped state. Cards: blocks 0–11. Sockets, back to front: attn (round), post-attn (square), ffn (hex), output (oval)Encoding meters, AG IFGtri | pSTG HG (Heschl): Δz at the tap, 0–0.10, pooled (paper Fig. 3)Level windows: mean Δz of the four meters at each of the card's tapsCentre-zero meters, ±0.015: attention-side (attn + post-attn) minus FFN-side (ffn + output) Δz, 95% CI; ▲ pre-specified directionToggle and knob: encoder and cohort; centre and all give the six-cell meanCard lamps: the cell's winning taps, semantic left, auditory rightLamp grids, Holm p < 0.05: auditory, the pre-specified test; semantic, the post hoc reverse-direction test (exploratory)Shared multilingual brain alignment in XLM-R and the NLLB-200 encoder is organized primarily around attention-side computations. The paper splits each sentence vector into a leave-target-out shared part (the mean of the other two languages' vectors) and an orthogonalized residual, then scores the shared part's encoding advantage Δz on fMRI from 94 listeners of Le Petit Prince at 12 blocks × 4 states. Attention-side minus FFN-side Δz is positive for the semantic regions in all 6 encoder × cohort cells, where the pre-specified hypothesis predicted the FFN side, and for the auditory regions in 5 of 6, 4 of them Holm-significant. Of the 12 winning taps, the highest-Δz block and state for each cell and region family, 9 are attn or post-attn, 3 are output and none is ffn.