I'm on the team at Embedl that built this, and wanted to share the numbers. What's in it INT4 weights, using our own quantization, about half the error of standard AWQ FlashHead, our faster LM head: the 262k-vocab output layer is a real cost at batch 1, so it only scores the likely candidates MTP head for speculative decoding Port to TensorRT Edge-LLM (0.10.1) underneath Speed (batch 1, 2,048-token prompts from real OpenAssistant conversations, 128 out, greedy, average of 20 requests) Decode: 39.6 tok/s End-to-end: 23.6 tok/s Prompt processing: 926 tok/s Peak memory: 24.8 GiB Accuracy vs the FP16 model (full test sets, the same engine as served) GSM8K: 96.4 (FP16: 96.1) IFEval: 83.5 (FP16: 81.9) MMLU-Pro: 79.1 (FP16: 79.5) For comparison: Jetson AI Lab reports about 35 tok/s decode with llama.cpp Q4_K_M + MTP. Prompt length isn't stated there, and they don't publish accuracy. To run it embedl infer Qwen/Qwen3.8-27B/thor --prompt "Hello" embedl serve Qwen/Qwen3.8-27B/thor (OpenAI-compatible API) The container download is about 1 GB, and the model weights are a separate download (~20 GiB). The full card has power, energy per token and methodology: https://embedl.com/Qwen/Qwen3.8-27B, and the ability to test the model on a Thor directly in the browser Let us know what model (and hardware) you want to see next!   submitted by   /u/No-Dragonfly6246 [link]   [comments]