Big update to the Delta Inference Engine, you guys asked for more features, more benchmarks and less clickbait. I'm here to deliver The short version: I've added almost everything requested and more. Delta Inference Engine now supports 64/96/128 GB of RAM and fully supports the new Unsloth quants. Unsloth IQ2 runs at the same 41 tok/s (192 GB RAM) using a new native reader for the Engram tables and some clever CPU kernel work. Compared to the old default model on my 99-question benchmark it went from 52 to 57 correct, against 59 for the full model. It's no full deepseek model quant, but they're a lot more usable than you might expect. Speed on one RTX 3090 24 GB (i5-13600K, 4x48 GB DDR5-5600, PCIe 4.0 NVMe, Windows, no WSL). Writing speed, tok/s, greedy, one chat at a time: RAM UD-IQ2_XXS, 2-bit, 301 GB UD-IQ1_S, 1-bit, 233 GB UD-Q2_K_XL, 2-bit, 335 GB UD-IQ3_XXS, 3-bit, 395 GB + the default 192 GB 41 41.0 31.3 28.2 128 GB 34.0 40.6 26.6 22.8 96 GB 29.3 36.5 22.9 18.3 64 GB 21.0 30.6 15.2 11.7 The 128, 96 and 64 GB rows are the same PC with the rest of the RAM locked away. Prompt reading: long prompts read at up to 740 tok/s at 192 GB and 600 at 128 GB (at ~64K tokens). A short prompt starts answering in about 2 s. Benchmark: 99 hard questions, greedy, 43K token budget: Set UD-IQ2_XXS (this release) ds4 Q2 (last release) Full model, API Qwen 3.8 Flash-Next, API Knowledge (40) 12 9 13 7 Maths (14) 10 10 10 8 Reasoning (30) 25 23 25 21 Coding (10) 10 10 10 8 Hard coding (5) 0 0 1 0 Total (99) 57 52 59 44 UD-IQ2_XXS is within 2 points of the full API, never scores below ds4 Q2 on any set while running at the same speed, and used 11% fewer tokens across the run. Agreement with DeepSeek's API on the next token, at the settings that ship: Model General text Code and maths Agentic UD-IQ2_XXS 91.1% 94.5% 95.9% ds4 Q2 89.9% 93.1% 95.1% UD-Q2_K_XL 92.5% 95.3% 97.0% UD-IQ3_XXS 92.7% 95.9% 97.4% UD-IQ1_S 89.1% 91.5% 95.3% For reference, two hosts of the full model agreed 98.3% and 98.9% in my test. By that measure Unsloth IQ2 is about 20% closer to the full model than the old ds4 Q2. If you want better quality: UD-Q2_K_XL (335 GB) runs 31 tok/s at 192 GB and 26.6 at 128 GB, at 92.5% / 95.3%. UD-IQ3_XXS (395 GB plus the default) runs 28 and 22.8 at 92.7% / 95.9%. What changed since last time: First token ~40% sooner: a ~180-token prompt went from about 3.8 s to 2.3 s. Agent turns read ~12% faster, and tool calls that low-bit models write slightly wrong are now read as tool calls. Unsloth quants at full speed: the engine reads their Engram tables directly. 64 and 96 GB are real options: SSD read-ahead (+22% at 64 GB), bigger expert memory, and prompt reads in file order (18-20% faster on low RAM). 128 GB: ds4 Q2 went from 33 to 36 tok/s. Default sampling: the drafter now samples its drafts too, about +3.5% at default settings. First launch shows which model suits your RAM, and you can pick any other or use your own. A Setup tab in the chat page changes model, speed vs quality and the second GPU. Public CI: the release zips are now built by GitHub Actions, so you can see how each one was made. EXL3: 2.5 bpw is on the menu and the GPU kernel is about 2x faster. In my tests UD-Q2_K_XL was both faster and closer to the API, so I still point people there. Basic multi-GPU: 7% uplift using a 3060 + 3090. It doesn't scale well to 3-4+ GPUs, so use Project Maya for those builds. Fixes: contexts past 65,535 tokens, in prose no longer ends the reasoning, no out-of-memory on the biggest files, a failed picture encoder no longer stops the model, faster prompts after a pause and many more. Support for custom model selection: just add them to the custom model folder and pick them in the web-ui Requirements: OS: Windows 10 or 11 GPU: NVIDIA RTX with 12 GB of VRAM or more, driver 580 or newer. 24 GB is the fast path, and AMD GPUs aren't supported yet RAM: 64 GB or more, 128 GB+ recommended Disk: a fast NVMe SSD, with 233-395 GB for the model (UD-IQ3_XXS also needs the default model) CPU: any x86-64 CPU, Intel or AMD. AVX2 is the fast path All of this is one PC. It needs testing on other hardware. Repo: https://github.com/musicurgy/delta-inference-engine MIT licensed. It's a Windows port of Xolotl, built on Strata. Thanks to DeepSeek for the model, Unsloth, nazeshinjite and ISTA-DASLab for the quants, and to the Xolotl and Project Maya communities, whose work and public numbers helped a lot. Claude wrote most of the code and I ran the project. If you try it, post your GPU, RAM, model and tok/s, good or bad.   submitted by   /u/musicurgy [link]   [comments]