TL;DR Basalt is the engine and it's doing the heavy lifting. Same model file, same box: about 3.5x llama.cpp's decode without drafting and ~5x with it, and ~7x faster prompt reads. Unsloth's new UD-IQ3_XXS (Dynamic v3.1, uploaded Nov 4) is the same size as the GSQ-RCO IQ3_S everyone uses on 16GB cards, but measurably more accurate: wikitext perplexity 3.382 -> 3.247, which basically closes the gap to BF16 (3.273). Costs 1 to 3.5% decode speed. Lamina (a further-trained MTP drafter by eddoursul) gets more drafts accepted than the stock Basalt drafter. Small but real speedup, zero quality impact since the main model verifies every token. Real agent coding session (opencode, 151 requests, up to 202K context): 195.6 tok/s average decode, 263 tok/s best request, 156 tok/s worst, ~5,700 tok/s prompt reads. Hardware: RTX 5080 16GB at its stock 360W limit (no overclock, no undervolt) Core Ultra 9 285K (8P + 16E), 2x32GB DDR5-7200 PCIe 5.0 x16, Fedora LAMINA Drafting: Basalt drafts with the model's own MTP layer. Basalt's upstream already ships a self-distilled version of that layer, and Lamina trains it further. It's a drop-in: same tensor layout, so the target model is byte-identical and only the drafter changes. What I saw: - drafts accepted per drafted token: 83.1% -> 85.8% on a replay of a real recorded session Lamina's README recommends temp 0.7 and max 3 drafts. I tested both: the 3-draft cap was 1 to 1.8% slower on tool-call-heavy work (long confident runs get cut off), and temp 0.7 changes what the model writes, so I kept the model's own sampling (1.0 / 0.95 / 20) and Basalt's draft settings. Basalt changes I made (all open PRs upstream) Host KV prefault: the first time a long conversation spills its KV cache to RAM used to stall 0.6 to 2.4s. Now ~75ms. Host KV pool on normal pages: on a fragmented box, transparent hugepage compaction caused 1.5 to 3.7s pins. Now ~23ms. Spin-then-yield in the decode loop's waits: about 1.3% less time per draft window. Expert start profile: the older v1 profile is 2% faster than the v0.2.0 one on this box. Plus a streamed conversation disk cache, --adapt-margin for expert swap hysteresis, elastic KV without RAM copies, and a bunch of timing diagnostics. Guide Branch with the exact engine (upstream v0.2.1 + my PRs) and a setup script: https://github.com/xyehya/basalt/tree/flashnext-5080-setup git clone -b flashnext-5080-setup https://github.com/xyehya/basalt && cd basalt setup/flashnext-ud-lamina/setup.sh bin/basalt serve --config basalt_config-flashnext.json --python .venv/bin/python --alias qwen3.8-flash-next It checks your machine, builds, downloads ~90GB (pinned revisions, every file SHA-256 checked), builds the UD-IQ3_XXS + Lamina model file and writes the config. You need an RTX card with, CUDA 13, 64GB RAM and ~180GB of free disk. If you already have the GSQ-RCO IQ3_S download, SHARD4_FROM= reuses its 28.8GB n-gram shard (it's byte-identical to Unsloth's once the shard number is rewritten). Huge thanks to jesdga95 for Basalt (support and star their repo), the Unsloth team for the UD quants, eddoursul for Lamina, ISTA-DASLab for the GSQ-RCO quants, and Qwhen 4? for the model.   submitted by   /u/yehyakar [link]   [comments]