We know from all the major chinese labs the current cutting edge architectures such as: conditional engram memory, hyper connections, attention residuals, sparse attention, etc... In the near horizon, we can likely expect looped transformers, Gated Delta Net 2 variants, maybe revival of diffusion transformers somehow but without the intelligence loss, and likely many others that I myself don't know of yet. What are they cooking?   submitted by   /u/Sadge404 [link]   [comments]