Just trying to figure out what to recommend to people running ESP32 voice frontends (xiaozhi-esp32, M5Stack AtomS3R, ESP32-S3 + I2S mic/speaker, WiFi to a local LLM). The wall every voice thread in this sub hits: 8B class (Llama 3.1, Qwen3.5 3B, Gemma): fast enough (40-60 tok/s on a 12GB card) but the conversation quality is bad. Repeats itself,loses the thread after 10 minutes, personality falls apart exactly where a companion needs it.70B+ quantized: smart, but 10-25 tok/s. A 30-token answer takes a second and a half of generation on top of STT+TTS, and the "thinking pause" kills the interaction. The usual answer is "go to gpt-4o-mini / Groq".The one local option I keep coming back to is Qwen3.8-Flash-Next(125B total, 6B active per token, 128k context) running through Strata on 12GB VRAM + 64GB RAM: 44-65 tok/s, OpenAI-compatible endpoint, so any ESP32 voice frontend just points at it. Caveats I already know: - preview / R&D build, deliberately under-trained - headline benchmarks are FP8, not the Q2/IQ3 quant that fits 12GB - needs 64GB system RAM, not just the GPU What I'd actually like to know from this sub: Has anyone run a 125B-class MoE locally as the brain of a voice assistant and measured end-to-end latency? Real numbers, not "it works".   submitted by   /u/myanimal22 [link]   [comments]