I am on the team behind the attached open-source research on novel approaches to attention - specifically we show: • You can sparsify the attention of Qwen 3.8 27B without observed loss on accuracy • Extending context up to 10M tokens loses accuracy on some specific tasks - but most, like NIAH is still fine (interesting insights) • You can natively train sparse attention on next-token prediction loss alone and achieve the same validation loss as full attention - but faster Hope the community likes it - while we focus on H200/B300 GPUs and larger scale system - I wanted to share our research here as well - as it can be applied to small open-weight models (making longer context cheaper on local hardware) Everything is fully open: kernels / SGLang integration / paper / details Hope you find it interesting / useful! (No hype - we try to be as transparent as possible)   submitted by   /u/Immediate_Sky_6566 [link]   [comments]