TL;DR: I built a small test harness that fixes real bugs in my home server scripts and configs, and ran a pile of local models against it one at a time, including some much larger ones. Then I took the two best mid sized models and had them debate each fix with one another across a few rounds. The debate scored 38 out of 44, against 31 and 30 for those two models on their own on the same setup, and it beat every single model I tried including a 120B. It won by solving cases that neither model got right alone. It did not do much for the confident wrong answers that pass a shallow check, which is the failure I care about most. This is all local, on two consumer cards. Has anyone else tried this, how did it go, and what did you learn? I have been building a little QA and self heal harness for my own infrastructure, and before I trust it to touch anything I wanted to know how good a local model actually is at fixing a broken script when the fix has to be right and not just plausible. I read in a few places that having two models argue tends to help, so rather than take it on faith I wired it up and measured it against everything else I had. The result surprised me enough that I wanted to ask around. The test suite The suite is 44 cases, each one a real bug I have actually hit on my own boxes. Home Assistant automations that fire when they should not, bash scripts that fall over under set -u, a compose file with smart quotes in it, a backup that quietly wipes itself when a mount is missing, that sort of thing. Every case ships a broken version, a visible check whose output the model is allowed to see, and a hidden test that the model never sees and that actually decides whether the fix is correct. The gap between those two is the whole point. A change that makes the visible check pass but fails the hidden test is what I call a false pass, and those are the dangerous ones because they look done. A handful of the cases are traps where the right answer is to make no edit and say the fault is elsewhere. The harness It runs on two GPUs, an RTX A5000 at 23GB and an RTX 3090 at 24GB, each with its own Ollama instance so that one model stays resident on each card and nothing has to reload between turns. The bigger models that do not fit get served through llama.cpp with their experts spilled into system RAM. Everything is asked one answer per case at temperature 0 with a fixed seed so a run repeats, and each proposed fix is applied in a throwaway sandbox where both the visible check and the hidden test are run on it. The single models tested Each of these got one answer per case across all 44, same prompt, same seed. Right is the hidden test passing (or a correct refusal on a trap case), false passes are fixes that satisfied the visible check but failed the hidden test. Model Size and quant Right False passes Gemma 4 31B (QAT) 31B dense, QAT int4, ~19GB 33 5 Qwen3.8 27B 27B dense, Q4_K_M, ~18GB 32 5 Qwen3.8-Flash-Next 125B (Strata) 125B MoE, 6B active, UD-Q4_K_XL, experts in RAM 31 6 Qwen3.8 27B (GSQ-RCO) 27B dense, IQ3_S, ~12GB 30 6 Nemotron 3.5 Super VL 120B 120B MoE, 12B active, UD-Q4_K_XL, experts in RAM 30 6 Nemotron 3.5 Bolt 74B 74B MoE, 9B active, ~22GB 28 5 Liquid LFM2.5 24B 24B MoE, 2.4B active, Q6_K 18 7 The thing that struck me is that the big models did not pull ahead. The 120B and the 74B and the 125B MoE all landed at or below two mid sized dense models I can hold in VRAM at the same time, and the bigger they were the slower they were by a wide margin. If you are GPU limited like me, that matters. The two I took forward were Qwen3.8 27B and Gemma 4 31B, one on each card, chosen from different families on purpose so they do not share the same blind spots. One note for fairness. The table above was measured on a slightly older Ollama. On the exact setup the debate runs on, a newer Ollama with one model pinned per card, those two on their own score 31 and 30 rather than 32 and 33, a small drop from the runtime change. The debate numbers below are on that same newer setup, so it is a like for like comparison. Two models debating In round zero both models propose a fix independently and the visible check runs on each. Then they take turns. Each one sees both proposals and what the check said about each, is told to look for concrete flaws in the other’s fix, to answer the objections raised against its own, and then to keep its position, adopt the other’s, revise with new edits, or abstain. This goes up to five rounds or until they land on the same change. I wrote the prompts to push hard against agreeableness, since the obvious failure mode is two models being polite and converging on nonsense. The pair scored 38 out of 44 with 5 false passes, against 31 and 30 for the same two models answering alone on the same setup. Digging into where the gain came from, it was almost entirely about six cases per run that neither model solved on its own but the discussion got right, while the debate almost never talked a model out of a correct answer it already had. I also ran a control where the two models shared their full private reasoning with each other versus sharing only their proposals, and it made no measurable difference, which I did not expect. It costs roughly six model calls and about five minutes of GPU per case. Moving forward The honest weak spot is the false passes, which barely moved, from six down to five. When both models land on a fix that passes the shallow visible check, the debate has nothing concrete to argue about and tends to just settle, and three cases in particular get waved through by both models every time. So the direction I am going now is to make the discussion argue from evidence rather than from rhetoric. Two changes. Agreement on a change that fails the check never counts as an answer, and a model cannot simply give in without something to back it up. And more interestingly, each model can propose a small test of its own that the harness actually runs on both fixes, so a claim about the other’s fix gets checked rather than just asserted. That run is going as I write this. I would really like to know if anyone else has gone down this road. Has anyone tried two models debating a fix like this? How did it go, and what did you learn?   submitted by   /u/Dodgy_Past [link]   [comments]