Hey r/LocalLLaMA, Blockway here, a small team from Hong Kong. After our v1 post, the most common question was basically "ok, but why does this model exist?" Totally fair. Here's our honest answer. The problem Long agent sessions eat VRAM. On a normal dense model every layer keeps a KV cache, and it grows with every token, so sooner or later you either cut the context or move to the cloud. We wanted long sessions to stay on your own GPU. So we built Volundr so that 54 of its 72 layers keep a small, fixed-size state instead of a growing cache. Why we wrote our own engine Existing servers aren't really built for that kind of model, so we wrote Agens Engine (MIT) around it. Two things it does that we're proud of: It remembers where a conversation left off, so a follow-up doesn't re-read the whole history. It stays inside a GPU memory budget you choose. Long context spills to system RAM instead of growing VRAM. The video One uncut take, real time. A 110K-token novel takes ~69 s to read cold. The follow-up question is answered in 0.5 s, because the engine just picks up where it left off. GPU memory peaks around 15.6 GiB. To be upfront: we recorded it on a 48 GB card with the engine held to a 16 GiB plan. We don't own a real 16 GB card yet. The best real-world numbers so far came from you. mmssix on HF tried it on a 24 GB card we'd never tested and got 86.4–106.7 tok/s. That's in the README now. Thank you! What's next We read every comment last time, and you were right about a lot: MoE: more of you could run it with an MoE variant. We're exploring one for a future version. Engram: the memory tables should be bigger. That's where we're heading. Image generation on the same GPU: because the LLM's memory stays flat, a 24 GB card has room left over. We'd like to run an image-generation model right next to it, with no model switching. It's a goal; we haven't built or tested it yet. Windows and Mac: Windows (starting with WSL2) is on our roadmap, and we're looking at Apple Silicon. A small box: we're also working on a compact desktop that runs all of this out of the box. We're a small team and there's still a lot to improve. If you give it a try, especially on a real 16 GB card, please tell us honestly how it goes. Every bit of feedback goes into training our next version. What we're hoping for is simple: a world where tokens are free, running on hardware you own. Engine: https://github.com/BlockWayz/agens-engine Weights: https://huggingface.co/collections/Blockway/agens-volundr-32b-v1   submitted by   /u/ComfortableKindly507 [link]   [comments]