I’m the developer of TensorSharp. I added MiniMax-M3.1-Flash support on Day 1 of its open-weight release, and wanted to share what’s working and some initial benchmarks for anyone looking to run the model locally. The biggest performance difference showed up with long prompts: 64K executor prefill reached 2,567 tok/s versus 262 tok/s in the tested llama.cpp branch, while server time to first token dropped from 293.4 seconds to 40.0 seconds. What’s supported The current implementation includes: GGUF inference, including MiniMax Sparse Attention with dedicated CUDA kernels. Text and image inputs, served through an OpenAI-compatible API. JSON/schema responses, tool calling, and multi-turn KV-cache reuse. Multi-GPU layer splitting and tensor parallelism. DSpark speculative decoding. CPU offload for routed MoE experts. I tested chat, long-prompt retrieval, structured output, single and parallel tool calls, an agentic tool loop, images, and concurrent requests. Hardware and comparison setup These measurements used 8× NVIDIA A40 GPUs and the same Unsloth MiniMax-M3.1-Flash UD-Q4_K_XL GGUF files for both engines. The model files total 232.3 GiB, before runtime allocations and KV cache, so this is a substantial multi-GPU setup. The CPU-offload results below also used this eight-GPU machine. For the baseline, I used Unsloth’s llama.cpp minimax-m31 branch at ef8f94cc, associated with PR #320. Upstream llama.cpp could not load this model when I ran the tests. The main comparison uses layer splitting on both engines, F16 KV caches, and no speculative decoding. Each engine ran separately with exclusive use of the GPUs. llama-server used flash attention and --no-kv-unified to keep its sparse-attention path active. Initial results Measurement TensorSharp llama.cpp support branch Executor prefill, 8K tokens 1,699 tok/s 1,312 tok/s Executor prefill, 32K tokens 2,449 tok/s 681 tok/s Executor prefill, 64K tokens 2,567 tok/s 262 tok/s Executor decode at 64K context 22.9 tok/s 15.4 tok/s Server time to first token, 64K prompt 40.0 s 293.4 s That’s 1.3×, 3.6×, and 9.8× prefill throughput at 8K, 32K, and 64K respectively. Decode gains are smaller. At 64K context, executor decode improved by 1.49×. For short-chat server requests, decode was almost identical: 26.3 versus 26.0 tok/s. Why the gap gets larger with context Beyond position 2,048, MiniMax Sparse Attention selects 16 blocks of 128 tokens per KV group. TensorSharp selects those blocks and runs block-sparse CUDA attention over them. The tested llama.cpp branch represents the selection using a dense mask and executes dense flash attention over the cache. That implementation difference becomes much more significant as the context grows. TensorSharp also pipelines prompt processing across the eight GPUs. Other configurations Tensor parallelism: TensorSharp’s --tp 8 reached approximately 41 tok/s for short-chat server decode, up from 26.3 tok/s with its layer-split mode. The agentic tool loop fell from 18.1 to 11.7 seconds. The tested llama.cpp branch rejects tensor splitting for this model, so this is a separate capability result. TP also had slower prefill and a smaller usable context on this hardware. DSpark: The main run averaged 32.3 versus 25.8 tok/s with the same drafter. TensorSharp used confidence gating to avoid expensive verification of weak drafts. With the gate disabled to match the baseline policy, the results were closer: 26.6 versus 25.8 tok/s. CPU expert offload: With --n-cpu-moe 24, approximately 79.7 GiB of routed-expert weights stayed in host memory. Short-chat decode reached 17.9 versus 13.8 tok/s. What still needs work Short-chat time to first token was slightly worse: 0.41 versus 0.39 seconds. Concurrent-request throughput improved in the main run, but per-request first-token latency was about 4% longer. These are early results from one host and one quantization. Most measurements use three repeats; the llama.cpp 64K executor prefill result is one run, while the 64K server result uses three repeats per engine. Validation includes tokenizer/template matching, synthetic-logit comparisons, and functional fixtures. There was an image-reading miss, and some generation tests hit token limits. Vision coverage is small, and native video temporal patches are not implemented. The measured TensorSharp build was an uncommitted development working tree, and raw run records are not included in the public report. Links TensorSharp on GitHub Unsloth MiniMax-M3.1-Flash GGUF Full benchmark report, configuration, and methodology If you’re testing M3.1 Flash locally, I’d be interested in your hardware, quantization, and prefill/decode results. Suggestions for additional baseline settings or useful workloads are welcome. I used an LLM to help draft and edit the English wording of this post.   submitted by   /u/fuzhongkai [link]   [comments]