Most of the speedup are touching along the likes like cutting the active params reletive to total params(MOE) but it does take a hit in stability & reliability compared to a dense-27b can be much more stable at low total params. We hade the innovation of MTP which is great and lots of other things like strata coming along but ever since MTP dense models havent gotten any architectural attention to speedups other than reduced reasoning tokens from community finetunes. with qwen4-27b coming up do we have any chance of any speedup breakthroughs?   submitted by   /u/Express_Quail_1493 [link]   [comments]