I am a GP in London, and I built OpenConsult, a research prototype that listens to a consultation and suggests questions and possible diagnoses as it goes. At the end of the consultation, OpenConsult writes a SOAP note and generates a diarised transcript. It is fully local; the doctor stays in charge, and every case is acted out or scripted. No real patients. In auto mode, it asks the patient questions itself, so the patient waits while the model thinks. I want the next question spoken within 3 seconds of the patient finishing. The machine: one RTX 4090 (24 GB), 96 GB RAM, i9-13900K. The speech model and an embedding model share the card with the language model. What I found so far: - MedGemma 27B (Q4_K_M) took 18 seconds per pass. Gemma 4 26B-A4B QAT took 5 seconds, with the same or slightly better clinical scores. - Ollama and llama.cpp were the same speed. vLLM was faster at 3.5 seconds, but it held 94% of the card against 70%, and the speech models need room too. - Sending the two calls of a pass together cut it to 3.8 seconds on Ollama and 2.8 seconds on vLLM. - Keeping one context for the whole consultation did not help. A fresh context at every call was at least as good. - I also tested Qwen3.6-35B-A3B, Nemotron 3.5 Lightning and gpt-oss-20b. Gemma 4 stays in front. A pass is two calls, one after another: an alarm check (about 1,000 tokens in, 160 out) and an assessment (about 800 tokens in, 570 out). Reading the prompt takes about 0.1 seconds per call. Almost all of the 5 seconds is the model writing its answer, at about 170 tokens a second. Speed is not the only test. The model must not drop a serious diagnosis from the list, and it must return clean JSON every time. My questions: What else would cut the time? Settings, speculative decoding, shorter answers? Which other engines should I benchmark? Which models should I try in place of Gemma 4 26B QAT? With 96 GB of RAM, could some models move to the CPU and system RAM, such as the embedding model or even the main model, and still answer fast enough? The project is at OpenConsult (AGPL-3.0).   submitted by   /u/looneycheetah [link]   [comments]