I’ve been building SonicBoom, a native tensor runtime written in C++/CUDA with a C ABI and no Python anywhere in the stack. I just tagged v0.0.1, and the first target is local LLM inference on consumer GPUs. Numbers (Gemma 4 E4B-it Q3_K_M on a laptop RTX 3050 8 GB, both runtimes using the same GGUF on the same machine, llama.cpp at commit 43fe9c64281): SonicBoom llama.cpp Decode (tg64) 51.65 tok/s 45.20 tok/s Decode @ depth 2048 47.13 tok/s 45.21 tok/s Prefill 4100 tokens 1051 tok/s ~1944 tok/s TTFT, 4000-token prefix + 100 new 503 ms (prefix reuse) Why it matters: the quantized matmul kernels (MMVQ/MMQ) are ported from ggml, so the kernels are the same ones llama.cpp uses. The 14% decode lead comes entirely from the layer above them. A graph planner rewrites the model before it runs: activations are quantized once and reused, norm and activation are fused with quantization, dead computation is removed, and each decode step is captured and replayed as a single CUDA Graph. Code derived from ggml is about 8% of the core. The rest is the runtime. Prefix reuse is exact: a continuation is token-for-token identical to a full prefill, which makes multi-turn chat and coding-assistant workloads cheap. What it can’t do yet: Prefill is about half of llama.cpp. It is a real batched path (int8 tensor-core plus cuBLAS), but there’s more to close. One verified model: Gemma 4 E4B. The quant formats are Q3_K, Q4_K and Q5_K only, so most Q4_K_M files from Hugging Face won’t load yet because they mix in Q6_K and Q8_0. Those formats are next, along with Qwen. It has only been tested on sm_86 (RTX 30-series), and sampling is greedy only. It’s not a drop-in llama.cpp replacement yet. The point of v0.0.1 is to show that the architecture works: once a graph compiler sits on top of the same kernels, the remaining gains come from the compiler instead of more hand-written kernels. Next on the roadmap are MLIR GPU codegen, speculative decoding, and more formats. GPL-3.0. Benchmark conditions, reproduce commands and the model sha256 are in the README: https://github.com/hardenedlinux/SonicBoom   submitted by   /u/nalaginrut [link]   [comments]