Tandem-vLLM is an expert-offload build of vLLM 0.31 for DeepSeek-V4.1-Flash. 0.1.1 ran only on Blackwell. 0.2.0 adds Hopper, Ada and Ampere GPUs, and two-GPU setups. Engine: https://github.com/ewhacc/tandem Model package: https://huggingface.co/ewhacc/DeepSeek-V4.1-Flash-1017-Tandem-VQ2 (293 GB) How it works The routed experts live in host RAM as 2-bit vector codes (VQ, 2.1 bits/weight). A GPU cache holds the most-used ones. Experts that miss the cache are computed by the CPU in parallel with the GPU. Nothing is skipped or substituted, so the output is exact for the package's weights. Everything else is stock vLLM: OpenAI-compatible server, continuous batching, prefix caching, tool calling, image input, DSpark speculative decoding. New in 0.2.0 GPUs: RTX 30xx / A6000 (Ampere), RTX 40xx / 6000 Ada (Ada), H100 / H200 (Hopper), plus Blackwell as before. Each kernel is checked against a reference at boot. Two GPUs, two ways: Tensor parallel (GPU=0,1): best with a strong CPU. Helper GPU (HELPER_GPU=1): the second card caches the hot experts the main card doesn't hold and computes them, so the CPU does less. Best with a smaller CPU or 24 GB cards. Needs no P2P or NVLink. Speed: one user, GSM8K, decode tok/s. The CPU matters as much as the GPU. GPUs CPU one GPU tensor parallel helper GPU 2× RTX PRO 6000 96 GB EPYC 9534, 64 cores 75.5 93.1 2× RTX A6000 48 GB EPYC 7F32, 8 cores 22.1 11.8 26.7 2× 24 GB (A6000s limited to 24 GB) EPYC 7F32, 8 cores 16.8 12.5 21.7 one GPU CPU tok/s H200 Threadripper PRO 7965WX 60.7 RTX 6000 Ada 48 GB EPYC 7F32, 8 cores 25.5 With 8 cores, tensor parallel is slower than one GPU because each GPU gets half of the CPU. The helper GPU keeps all of the CPU for one main card. The 2× 24 GB row is the 3090/4090 memory shape, run on A6000s limited to 24 GB. Real 4090 numbers are coming. A 24 GB card is the minimum main GPU. 16 GB cards don't fit as the main card. At 32 streams, one RTX PRO 6000 serves 414 tok/s in total. Quality vs the official FP4 checkpoint (greedy, thinking low) official VQ + FP4 top 52 VQ only GPQA-diamond 81.3% 76.8% GSM8K-200 192 194 passkey 8k / 32k / 94k 9/9 9/9 The 2-bit experts cost GPQA accuracy; easier tasks and long-context retrieval don't show it. FP4_TOP=52 keeps the 52 most-routed experts per layer in original FP4 and wins back about half of the loss. What you need 300 GB of disk for the package, and 128 GB+ of RAM (256 GB to pin everything). Linux, CUDA 13 toolkit, an AVX2 or AVX-512 CPU. One installer script, its own venv, no Docker. Built on vLLM and vLLM-Moet. Tandem is a fork of vLLM-Moet's expert-offload engine, ported to vLLM 0.31 / DeepSeek-V4.1. Numbers from other boxes are welcome, especially real 3090s and 4090s. P.S. For one user on a 16-32 GB card with 64-128 GB of RAM, our llama.cpp-based sister project tandem.cpp is faster (33 tok/s on a 5090 + 128 GB).   submitted by   /u/smflx [link]   [comments]