Most LLM inference benchmarks focus on tokens/sec or TTFT. But faster inference doesn't mean much if the output quality degrades, especially with aggressive quantization, speculative decoding, or very long contexts. I ran a comprehensive benchmark of Qwen3.8-27B across six inference engines, comparing not just speed, but also output quality, long-context performance, and different model formats. The results are summarized in the attached infographic. https://preview.redd.it/sh0tv3i3rvzh1.png?width=1600&format=png&auto=webp&s=4be79ec6ea76212685310eb7ef1199f344482a7f What's different about this benchmark? 1. Speed AND quality — not just tokens/sec We measured: Time to First Token (TTFT) Prefill throughput (tokens/sec) Decode throughput (tokens/sec) Concurrent request performance End-to-end latency across different tasks Output quality using automatic checks The goal was to understand whether performance optimizations actually preserve model quality. The quality comparison is particularly important when evaluating quantized models and speculative decoding. 2. Long-context testing: 64K, 128K, and 256K tokens Many benchmarks use relatively short prompts, which don't reflect real-world workloads like large codebases, document analysis, or long-running agents. We tested three long-context configurations: 64K tokens: Performance across different inference and concurrency configurations 128K tokens: Prefill, decoding, and memory behavior under larger workloads 256K tokens: Stress-testing long-context inference and identifying practical limitations The results reveal substantial differences between engines as context length increases. Some configurations that perform well with shorter prompts become much less competitive at 128K–256K. 3. GGUF vs NInfer model formats Another interesting aspect is that this isn't exclusively a GGUF benchmark. We included both GGUF-based and NInfer-based model configurations, allowing us to examine how different model representations and inference stacks perform under comparable workloads. This is especially relevant for people choosing between lightweight local inference setups and more specialized inference engines. One important caveat: these comparisons reflect complete engine/model configurations, not an isolated measurement of the model format itself. 4. Real-world task performance Synthetic throughput numbers don't tell the whole story. We also measured end-to-end latency across multiple task scenarios, including coding, reasoning, and agent-style workloads. For practical applications, completing the task faster often matters more than achieving the highest peak tokens/sec. Key takeaways There is no universal winner across every workload. Decode throughput, TTFT, and end-to-end latency can produce very different rankings. Long-context performance changes the competitive landscape considerably. Quality validation should be part of inference benchmarking, not an afterthought. Comparing GGUF and NInfer configurations provides another useful perspective beyond conventional GGUF-only benchmarks. Disclosure I'm the author of TensorSharp, one of the engines included in this comparison. The benchmark also evaluates competing inference engines, and the goal is to provide useful data for the local LLM community rather than claim that one engine wins everything. I would especially appreciate feedback on the methodology, quality measurements, and long-context testing. Questions for the community: When choosing a local inference engine, do you prioritize TTFT, decode speed, or end-to-end task latency? How important is quality validation when comparing quantized models? Are 128K–256K contexts becoming part of your actual local workloads? Full benchmark results are in the attached infographic. TensorSharp: https://github.com/zhongkaifu/TensorSharp Would love to hear what additional engines, quantization formats, or workloads you'd like to see benchmarked next.   submitted by   /u/fuzhongkai [link]   [comments]