I am curious what people are currently using as the core model for their local AI agents (e.g., in Hermes or similar setups). I've was initially running Qwen 3.6 35B-A3B IQ3 XXS on hermes working off (Intel Core Ultra 5 245K · Corsair 64GB DDR5 RAM · Asus TUF GeForce RTX 5070 Ti 16GB GDDR7 OC Edition Graphics Card running ai inferance on local home lab running UnRaid OS). First started with ollama years ago (where i was running 9B and 14B fully in vram) and months back switched to llama.cpp, then to llama-swap to be able to hot swap models and use MoE models. I started off getting 40 tokens per second (more n-cpu-moe offload and less vram use), then tried the MTP version for 55ish. Learning more about finding the balance between n-cpu-moe offloading and finding the sweet spot for VRAM, i got that to about 60t/s. Then I discovered fine tunes like Swift (55t/s) and Nail (65t/s). Then endless frontier BigBangv1 which gave a blazing 100 t/s. I was running that and saw people mentioning they use the dense Qwen3.8 27B, but it seemed slower at 45t/s. I have very rough tested (only speed) base Strata with Qwen3.8 Flash Next 125B 1Q2_XS which gave 90t/s but used 95% VRAM and 95% RAM which wasnt sustainable (ill likely try it again in future but for now reverted back to llama-swap) Learning more like how agent workflows are very dependent on prefill speed and prompt processing alongside tokens/sec, then LFM has crazy prefill and t/s but its quality it lower, I am not sure what is the right balance! But also how context length is important for agents, and the k v cache of the context too! I settled on not going any lower than k v 8, aiming for 128k context but 90k is acceptable if the model is specialised. I was manually guessing how all these quants and CPU offload configs (n-cpu-moe) scaled under heavy context. I also wanted to know if the same ai model actually performed better on agentic tasks for a Q3 vs Q4, is the drop in speed (generate t/s or prefill t/s) justifiable for better quality etc. So I begin testing out a bunch of fine-tunes and MoE variants to dial in different profiles. I’ve been cycling through options like Empero, BigBang, and Ornith (tried Swift and Nail too, but ended up dropping them). Generally, I lean toward the MoE variants over the dense Qwen 3.8 27B because of how they handle things on consumer hardware (testing shows they are slower, dont provide any significant boost in quality, require lower context, use more VRAM and gpu power etc). My use case isnt any heavy coding, just small scripts and having hermes act as a personal assistant which will evolve over time. Then with hermes I was using endless-frontier_BigBang-v1-35B-A3B-IQ3_XXS with 100 t/s, prefil 2359 t/s, 88% VRAM usage which was perfect. I originally wasn't thinking about its 'quality' since this was a massive step up from the original Qwen 3.6 35B-A3B IQ3 XXS. Looking forward to a qwen4 release to test soon! I'm looking at setting up specialized profiles on hermes, so far i am thinking Cyber-Tiel-Coder-35B-A3B-MTP-UD-IQ3_XXS the dedicated coder profile. Might test out GrootTeam_Groot-19B-A3B-261023-Q4_K_M as the main general profile. Otherwise Ornith is looking good. Or Empero or BigBang. I have fallbacks to openrouter/Nvidia 550B, then /free I ended up hacking together a simple local script just to measure t/s and prefill, but it kind of snowballed... I added some core eval tests (accuracy, reasoning metrics) and turned it into a small dashboard that spits out a weighted "agent score" to help me rank them objectively. Still testing these models on more rows of the HumanEval, MMLU-Pro, MBPP, tool calling and long context tests, but it takes a lot of time. Images attached for reference from my latest tests. Running these tests takes a long time and for local use cases I dont need that margin of error from a full test across all 12000 MMLU-Pro questions, so i have been further developing a method to spend less time on testing models which drop lower on the rankings, adjusting smaller runs with high uncertainity etc, its till a work in progress but its progressively been giving me models to consider for the top few I actually want to use in the massive amount of options on hugging face that can run on my hardware. Even with the dashboard telling me what the numbers say, it's still surprisingly tough to pick a definitive daily driver. I would appreciate any input or suggestions. Feel free to look at the interactive html i put on github. You can see all the test on the models I have run. https://htmlpreview.github.io/?https://github.com/Maximo101/Local-LLM-Benchmark/blob/main/assets/5070ti_benchmark_dashboard.html Questions I am unsure of; for those who have more experience with agents on local hardware: ------------------------------------------------------------------------------------ 1. Which models have won you over for agentic use (hermes, openclaw etc)? 2. How heavily do you weigh prefill vs. generation speed, quantisation vs quality, VRAM % vs context (quality and size) when choosing AI Models? What sort of profiles do people use in agentic setups like hermes? How many profiles do you use. (If anyone is fighting similar benchmarking battles for their local endpoints, I cleaned up that testing suite and open-sourced it here: https://github.com/Maximo101/Local-LLM-Benchmark) I am not trying to promote this, i built it for my own purposes, but decided to share it in case it is useful for anyone else.   submitted by   /u/maximo101 [link]   [comments]