v0.3.0 of NInfer for Turing is out, and this is all about streamlining the UX. For context, this is an engine for running Qwen3.6/8-27B (and ONLY Qwen3.6/8-27B) on a single modded 22 GB RTX 2080 Ti. It is designed to ONLY use VRAM, no system RAM offloading, hence it does not support 35B-A3B. Fine-tunes and abliterations etc are supported as well, just use the .ninfer checkpoint converter. What's new - Native Windows. Previously this required Linux, WSL2, or Docker. Now Windows users don't need to worry about partitioning/releasing resources (WSL or Docker instances require system RAM allocation even if your use case inside them doesn't). - Prebuilt releases. You can still compile it yourself, requiring CUDA, Visual Studio and FFmpeg and takes a while, but now you can also just download a pre-compiled zip and run a single script. - Streamlined. Removing "useless" (for us) code paths for RTX 5090, FP8, FP4, and 35B-A3B (fits fully in 32GB VRAM but not in 22GB) cut the built binaries from 1.56GB to 0.28 GB. Builds now take minutes, not hours. Outputs remain bit-identical. What you get - OpenAI-compatible and Anthropic-compatible APIs, plus a command-line tool. - Image and video input, thinking mode, and CUDA Graph decode. - 64k context by default, using an INT8 KV cache. The weights take 15.4 GiB, so you need the 22 GB card. - MTP speculative decoding, on by default. Numbers (one 2080 Ti 22 GB, Windows 11, single request) Test Result prefill, 512-token prompt 616 tok/s prefill, 64k-token prompt 458 tok/s decode, no MTP 27.0 tok/s decode, MTP, real code prompt 47.5 tok/s (26.3 without MTP) decode, MTP, real prose prompt 28.8 tok/s (26.0 without MTP) (Apache-2.0, derived from Neroued/ninfer)   submitted by   /u/xrailgun [link]   [comments]