Qwen3.8-Flash-Next NVFP4 on one RTX 5090 - prefill and decode against context Full 262,144-token context, one request at a time, MTP speculation on, int8 KV, identical test for all three engines. Every number is tokens/second. Prefill = prompt tokens / time to first token. Decode = generated tokens / decode time. The fastest engine in each row is in bold. Each data point is a fresh, non-repeating prompt: the servers reported zero prefix-cache hits. Item Value GPU NVIDIA RTX 5090, 32 GB CPU, RAM Intel Core Ultra 9 285K, 247 GB DDR5 Model Qwen3.8-Flash-Next, NVIDIA NVFP4 experts, 48 layers, 512 experts Artifacts basalt Qwen3.8-FN-NVFP4.basalt, capy-flash qwen3_8_flash_next_nvfp4_idxgdnq8.ninfer, ninfer-ext qwen3_8_flash_next_nvfp4_draft.ninfer Context 262,144 tokens, swept in 16k steps (16 points) Output 256 tokens per request, ignore_eos Builds basalt fe7fd96f, capy-flash 4e45286a v0.7.1, ninfer-ext 3241c511 Engine configuration (after tuning) Setting basalt capy-flash ninfer-ext max context 262,144 262,144 262,144 KV cache int8, elastic int8, elastic int8 expert cache auto auto (elastic) auto speculation MTP, spec window 7 MTP, 3 drafts MTP, 3 drafts CPU side pool-workers 15, ple-io ram miss-split auto, 16 CPU threads ngram mapped VRAM used 27.1 GiB + 40.7 GiB host RAM 28.3 GiB + 65.2 GiB host 23.1 GiB Greedy decoding, temperature 0 Prefill, tokens/s Context (tokens) capy-flash basalt ninfer-ext 16k 6,921 8,490 4,473 32k 7,736 8,712 5,193 48k 8,015 9,211 5,764 64k 7,976 9,052 6,113 80k 7,806 9,206 6,269 96k 7,879 8,949 5,842 112k 7,800 9,061 5,353 128k 7,677 8,978 5,696 144k 7,238 8,842 5,321 160k 7,429 8,773 5,311 176k 7,391 8,670 5,182 191k 7,283 8,553 5,187 207k 7,207 8,511 5,057 224k 7,063 8,434 4,847 239k 6,912 8,347 4,846 255k 6,828 8,285 4,748 Decode, tokens/s Context (tokens) capy-flash basalt ninfer-ext 16k 198 242 92 32k 186 265 96 48k 170 251 106 64k 160 246 99 80k 167 242 80 96k 157 205 87 112k 157 225 90 128k 165 216 88 144k 157 202 88 160k 151 187 87 176k 151 199 78 191k 162 196 83 207k 152 185 76 224k 151 187 74 239k 144 177 73 255k 138 176 75 Sampled decoding, temperature 1.0, top_p 0.95, top_k 20 Prefill, tokens/s Context (tokens) capy-flash basalt ninfer-ext 16k 6,881 8,571 4,198 32k 7,751 8,639 5,082 48k 8,046 9,096 5,693 64k 7,971 8,875 6,089 80k 7,782 9,200 6,263 96k 7,806 8,928 5,805 112k 7,625 9,050 5,354 128k 7,598 9,048 5,714 144k 7,171 8,773 5,347 160k 7,326 8,879 5,309 176k 7,384 8,749 5,191 191k 7,179 8,570 5,185 207k 7,192 8,375 5,053 224k 7,102 8,333 4,848 239k 6,999 8,252 4,850 255k 6,879 8,141 4,753 Decode, tokens/s Context (tokens) capy-flash basalt ninfer-ext 16k 178 235 79 32k 149 212 87 48k 164 236 99 64k 158 242 85 80k 150 227 69 96k 137 187 84 112k 132 186 87 128k 130 207 83 144k 145 203 84 160k 133 178 72 176k 155 214 64 191k 150 169 77 207k 161 178 77 224k 145 178 57 239k 139 146 69 255k 143 163 65 Summary across all 16 points Engine Prefill tok/s Decode greedy tok/s Decode sampled tok/s Sampled vs greedy decode basalt 8,285 - 9,211 176 - 265 146 - 242 -7.1% capy-flash 6,828 - 8,015 138 - 198 130 - 178 -7.7% ninfer-ext 4,473 - 6,269 73 - 106 57 - 99 -9.8% MTP draft acceptance, mean over the 16 points Engine Greedy Sampled Change basalt 0.74 0.67 -0.08 capy-flash 0.57 0.49 -0.07 ninfer-ext 0.56 0.44 -0.12 Headline basalt is fastest at every context depth in both modes: 8.1k-9.2k tok/s prefill and 176-265 tok/s decode greedy. capy-flash is second: 6.8k-8.0k tok/s prefill, 138-198 tok/s decode greedy. ninfer-ext is third: 4.2k-6.3k tok/s prefill, 73-106 tok/s decode greedy. Decode ranks exactly with MTP draft acceptance: basalt 0.74, capy-flash 0.57, ninfer-ext 0.56 under greedy. Sampling costs 7-10% decode for every engine and nothing in prefill. Tuning that changed the result Engine Change Decode at 33k / 131k (tok/s) basalt stock auto picked pool-workers 1 on this CPU 122 / 114 basalt pool-workers 15 238 / 200 basalt plus ple-io ram 248 / 228 capy-flash stock profile 175 / 138 capy-flash plus miss-split auto and 16 CPU threads 184 / 173 ninfer-ext int8 KV (nvfp4 KV measured slower) 106 / 83   submitted by   /u/giveen [link]   [comments]