At EMNLP 2026, an A* conference currently taking place in Hungary, researchers from T-Tech presented LIMe, a method that makes it possible to train language models with approximately 15% less compute without compromising quality. In traditional Transformers, each layer relies solely on the hidden state of the previous layer, meaning all contextual information must pass through it. The authors demonstrate that this creates pressure toward representation collapse: tokens become increasingly similar to one another, their distinguishing features fade, and model quality suffers as a result. LIMe (Layer-Integrated Memory) is a lightweight extension of the attention mechanism. It leverages the key-value buffers already maintained for each layer and introduces learnable routing weights specific to each attention head and layer. These weights allow attention heads to retrieve and combine representations not only from the immediately preceding layer but also from earlier layers. The hidden state size remains unchanged, while previously computed information is reused. In experiments with models containing approximately one billion parameters, LIMe achieved the same training loss as baseline LLaMA while using 15% less compute. The method also outperformed competing approaches on language modeling and multi-step reasoning benchmarks. The most significant improvements were observed in synthetic arithmetic tasks and architectures with greater depth. The authors also report that LIMe produces higher entropy in value vectors and better token separability, suggesting that it effectively mitigates the underlying representation collapse problem. Analysis of the learned routing weights further reveals that the model systematically reuses both local and long-range features. Paper: https://arxiv.org/abs/2502.09245 Official implementation: https://github.com/corl-team/lime   submitted by   /u/ChampionshipLimp1749 [link]   [comments]