Hey all, I wanted to share that we're working on zml/llmd, a fast (we believe the fastest) inference engine that currently runs on NVIDIA, AMD, Intel, TPU and Metal. It has all the features of a modern LLM server: continuous batching, prefix-caching, speculative decoding (DFlash, DFlash2 and DSpark), tool calling, tensor and experts parallel automatic sharding, native VFS with s3:// and hf:// support streaming straight to GPU. The currently supported models are: Qwen 2, 3, 3.5, 3.6 and 3.8 series (dense and MoE) Gemma 3 and 4 series (dense and MoE) LFM2.5 series Mistral 3 and Ministral series LLaMa 2 and 3 series Poolside's Laguna S 2.1 DeepSeek 4.1 Flash Muse Glimmer Qwen3.8 Flash Next Xiaomi MiMo V2.6 Mistral Large 4 GLM 5.2 GLM 5.3 LFM 3 Beam 1.0 Ripple 65B Our images are a lot smaller than alternatives: Platform Image Size CUDA zmlai/llmd:cuda 1.96 GB ROCm zmlai/llmd:rocm 4.77 GB TPU zmlai/llmd:tpu 272 MB OneAPI zmlai/llmd:oneapi 454 MB Metal brew install zml/zml/llmd 142 MB We just came out with a new release today: zml/llmd llmd-20261106.0 and we'd love to get your feedback on it!   submitted by   /u/siwu [link]   [comments]