Part 2 of my agent test (part 1 was a 4B model) Hardware I used: MacBook Pro M5 Max, 64 GB. Qwen 3.8 27B Q4 on llama.cpp, thinking off. Setup same model and server for everyone 5 small coding tasks, each with a little trap each agent locked in a sandbox: it can only touch its own folder and only talk to the model auto-approve on, telemetry off, 5 min limit Results # Agent Passed Median time Median tokens Cache reuse Tried to go online 1 Pi 5/5 38 s 9.2k 72% no 2 DeepSeek Harness 5/5 60 s 35.3k 83% no 3 Codex CLI 5/5 94 s 36.3k 79% every run 4 OpenClaw 5/5 106 s 39.0k 72% unclear 5 Cline 5/5 152 s 29.6k 79% every run 6 OpenCode 4/5 50 s 32.4k 75% every run 7 Hermes 4/5 80 s 64.9k 93% no 8 Atomic Agent 5/5 56 s 30.4k 86% no 9 goose 4/5 98 s 20.3k 76% every run 10 Qwen Code 4/5 183 s 107.9k 74% unclear Tokens and cache Pi is crazy lean: about 9k tokens per task, the lowest on every single task. Qwen Code was the hungriest at about 108k per task. Atomic Agent is the odd one out in how it works. It was the only one talking to llama.cpp's native endpoint with a grammar that forces the tool calls into shape. The other nine all went through the OpenAI-style API. It's also built to keep its prompt the same from step to step, which is why it reused 86% of the cache (2rd of 10). It stayed offline the whole time and passed 5/5. DeepSeek Harness was the surprise. It’s still an alpha, but it went 5/5, never tried to go online, and had the best cache reuse after Atomic Agent. The trap and the network The log task got three agents. They grepped "ERROR" and got 244 instead of 64, because some normal lines just mention errors. Codex, Cline, goose and OpenCode tried to reach the internet in every run, even with telemetry off. The sandbox blocked it, so I don't know what they were calling. OpenClaw and Qwen Code are "unclear" because they run as plain node and I couldn't tell them apart. llama.cpp notes Codex needs llama-server's /v1/responses now Hermes needs at least 64K context Cline and Qwen Code have usage stats on by default, DeepSeek Harness uploads session logs (per their docs). I turned it all off Which column matters most to you when you pick an agent for a local model like Qwen 3.8 27B: passing, speed, tokens, or staying offline?   submitted by   /u/latent_lurker8587 [link]   [comments]