I'm now getting 3x the speed than with llama.cpp (Unsloth fork): 10.52.637.962 I slot print_timing: id 0 | task 0 | prompt eval time = 620987.80 ms / 38761 tokens ( 16.02 ms per token, 62.42 tokens per second) 10.52.637.966 I slot print_timing: id 0 | task 0 | eval time = 15443.97 ms / 292 tokens ( 53.07 ms per token, 18.84 tokens per second) 10.52.637.967 I slot print_timing: id 0 | task 0 | total time = 636431.76 ms / 39053 tokens Strata (same Hermes session, loaded with Strata and a turn after the previous one with llama.cpp): [strata] reading the prompt: 8,192 of 38,968 tokens, 5 s so far [strata] reading the prompt: 16,384 of 38,968 tokens, 9 s so far [strata] reading the prompt: 18,720 of 38,968 tokens, 11 s so far [strata] reading the prompt: 26,912 of 38,968 tokens, 16 s so far [strata] reading the prompt: 35,104 of 38,968 tokens, 20 s so far [strata] reading the prompt: 38,963 of 38,968 tokens, 23 s so far [strata] thinking: 2 of max 92096 tokens, 36.7 tok/s, 24 s [strata] thinking: 48 of max 92096 tokens, 44.8 tok/s, 25 s [strata] thinking: 94 of max 92096 tokens, 44.7 tok/s, 26 s [strata] thinking: 144 of max 92096 tokens, 46.1 tok/s, 27 s [strata] thinking: 212 of max 92096 tokens, 51.2 tok/s, 28 s [strata] thinking: 273 of max 92096 tokens, 52.8 tok/s, 29 s [strata] thinking: 342 of max 92096 tokens, 54.8 tok/s, 30 s [strata] thinking: 407 of max 92096 tokens, 56.0 tok/s, 31 s [strata] thinking: 463 of max 92096 tokens, 55.9 tok/s, 32 s [strata] thinking: 519 of max 92096 tokens, 55.8 tok/s, 33 s [strata] thinking: 578 of max 92096 tokens, 56.1 tok/s, 34 s [strata] answering: 651 of max 92096 tokens, 57.5 tok/s, 35 s [strata] answering: 716 of max 92096 tokens, 58.1 tok/s, 36 s [strata] answering: 769 of max 92096 tokens, 57.6 tok/s, 37 s [strata] done: 799 tokens in 37 s (57.4 tok/s) (stop, cancel=False), expert cache 94.5% hit (+1.3% of the routed experts over PCIe) For PP llama.cpp took 10 mins, Strata 23 seconds, and I get 3x the TG speed... I have no idea how is that possible, but I've been using it for about 10 days, every day and results are consistent (actually latest updates made it faster, because I was getting "only" about 45 t/s TG). And I load both with the very same files: qwen3.8-flash-next-unsloth-ud-q4_k_xl and full KV (fp16) and 130k context (110k with llama.cpp), and even with "--kv-resident", "32768" speed barely drops above that. I love llama.cpp and I'm starting to love Strata as well! Thank you Strata!   submitted by   /u/relmny [link]   [comments]