So I've been reading a lot about "engines lobotomizing the models they run", either because they quantize already quantized weights even further, or because they automatically turn on lossy flags to gain decode/prefill performance. I wanted to test this out by myself, not only with KLD but with real tasks, so I set up an experiment using Qwen 3.8 Flash Next with Basalt (disclaimer: I'm the developer) and two of the most common lossy flags: - Routing bias: the engine pushes the model toward experts that are already on the GPU, so it sometimes gets a convenient expert instead of the one it actually wanted. - Tail skip: when one of the model's lower expert picks isn't on the GPU, the engine just skips it instead of fetching it. Both are faster because fewer experts have to come over from RAM, but both change the model's output. I added them to the engine as opt in flags so I could measure exactly what they cost with everything else fixed, only changing the flags. DISCLAIMER: This post and the experiment were run by me, but it has been polished by AI. The rig: RTX 5090 (400 W) + RTX 5060 Ti (180 W) and 128 GB of DDR5. Settings tested: - Shipped: bias 1.5, skip rank ≥ 7. Values other engines ship (or have shipped) as defaults on smaller cards. - Worst: bias 16, skip rank ≥ 1, plus relaxed draft acceptance. Worst case scenario, not something anyone ships or would make sense to use. 1. KLD 100 chunks of wikitext, decode path, against a llama.cpp Q8_0 reference. Exact IQ3_XXS and Q2_0 are in there for scale. Pack + setting File PPL KLD mean Same top-1 Q8_0 exact 178 GiB 3.276 0.031 94.1% NVFP4 exact 122 GiB 3.304 0.068 91.4% IQ3_S exact 81 GiB 3.385 0.155 87.3% IQ3_XXS exact 74 GiB 3.470 0.243 84.0% IQ3_S + shipped 81 GiB 3.583 0.244 84.3% NVFP4 + shipped 122 GiB 3.931 0.348 81.8% Q8_0 + shipped 178 GiB 4.221 0.401 79.4% Q2_0 exact 65 GiB 3.949 0.436 78.2% NVFP4 + worst 122 GiB 5.917 0.823 71.3% Q8_0 + worst 178 GiB 12.858 1.616 56.8% Note: even exact Q8_0 is 0.031 away from the reference. That's mostly the reference's own noise, not damage: llama.cpp's Q8_0 rounds a few spots of this model more than Basalt does, which adds about 0.01-0.02 to every row equally, so the rows still compare fairly. With the shipped values every pack lands between IQ3_XXS and Q2. The bigger the pack, the further it falls, since more of it sits outside VRAM and gets rerouted: a 178 GiB Q8_0 ends up next to the 65 GiB Q2_0. 2. Does it matter on real tasks? KLD can overstate what you'd notice, so I also ran a task eval. Thinking off, model-card sampling (temp 1.0, top_p 0.95, top_k 20), seeded, fresh server per config. Ladders: 300 ladders of chained steps, rules held at once and reversed digit copy. Score = % of rungs passed. Facts: 1,200 closed-book questions × 3 seeds from PopQA (5 popularity bands, rare to famous) plus SimpleQA. Score = % correct. Each delta is against the same pack run exact, with a 95% bootstrap CI. Config Ladders Facts Ladders Δ Facts Δ Prose tok/s Q8_0 exact 45.4 38.1 117 NVFP4 exact 45.6 38.0 272 Q8_0 + shipped 43.8 36.6 -1.6 [-3.4, +0.2] -1.5 [-2.5, -0.5] 202 NVFP4 + shipped 44.1 36.8 -1.5 [-3.2, +0.2] -1.2 [-2.2, -0.4] 271 IQ3_S exact, small card* 44.1 36.7 192 IQ3_S + shipped, small card* 38.9 34.3 -5.2 [-7.1, -3.2] -2.3 [-3.4, -1.3] 299 Q8_0 + worst 26.6 2.7 -18.8 -35.3 379 * Small card: the 5090 alone with the expert cache capped at 5 GB, to mimic a 16 GB card. Bold = CI excludes zero. Experiment was run this way because the throttled 5090 is still much faster than the 5060 Ti. What it says: With plenty of VRAM, the shipped values cost about 1.5 points (≈4% of the facts the model knew) and buy little: nothing on NVFP4 (272 → 271 tok/s), more on Q8_0 (117 → 202), which is still slower and worse than NVFP4 exact. On a small card they buy real speed (+56% prose) and cost about 3× more. Ladders drop 5 points and facts 2.3, both clearly. The worst setting gives confident nonsense. Facts fall from 38% to 3%, with answers like "The Great Composer". Some tests barely see it. Rules held at once moved only -3.8 even at the worst setting, and an earlier, easier battery couldn't rank the configs at all. The damage shows up in chained reasoning and recalled facts, which is what the rerouted experts hold. What it looks like Same prompt ("Write an essay on the Odyssey"), Q8_0 with the shipped values: polished, well argued, and wrong in a dozen places. Phemius is a woman, Helen sneaks into Troy, Odysseus's real name means "Nobody", and a bard called "Theso-mys" gets killed mid-verse. Names and rare words break first. With the worst setting it degrades into "the n of n" and loops. Bonus: a loop that's the model's own Not every bad output is the engine's fault. One coding prompt (the model checks a quantity parser and writes a leading-zeros example) sends Qwen 3.8 Flash Next into an endless run of "0" in its reasoning: Setup, same request, seeds 1-4 Stuck in zeros Basalt, NVFP4 4 of 4 Basalt, IQ3_S 4 of 4 Basalt, Q8_0 3 of 4 llama.cpp, same IQ3_S GGUF 3 of 4 Official API (OpenRouter pinned to Alibaba) 6 of 6 that returned The odds of another "0" are the same on every pack and both engines: 0.99 after 10 zeros, and 95% or more at every step past about 100, so top_p 0.95 leaves "0" as the only choice. It still happens at temperature 1.0. The official API cuts the run at about 500 zeros with "......" and lets the model answer, so you never see it there. (I can't confirm the API serves byte-identical weights, but it loops the same way.) Basalt now does the same thing in the open: after 256 identical tokens in the reasoning it closes the thinking block, lets the model answer, and logs that it fired. It only ends the loop; it doesn't change any token the model picks. Caveats One rig, one model. Ladders ran one seed, facts ran three. KLD is against a llama.cpp Q8_0 reference, so even exact Q8_0 is 0.031 away. Speeds are single cold runs, for scale only. Basalt ships these flags off by default and prints a warning when you turn them on. The flags are in the public repo, so you can rerun the KLD numbers yourself. TLDR: there's no free lunch. Engines can buy speed on MoE models by turning on speed "hacks". With values some engines ship, every pack drops to somewhere between IQ3_XXS and Q2 on KLD. A Q8 ends up next to Q2. This is as bad as it gets. In practice the model forgets things it knew and loses the thread on multi step problems. Small cards get the biggest speedup and pay the most for it. If your engine is faster than it should be on a model that doesn't fit, check whether it's still running the model you downloaded.   submitted by   /u/jesdga95 [link]   [comments]