I’m running a sandbox simulation with a fleet of 50-100 agents on OpenClaw, and to minimize LLM calls, I use a decision model to judge whether an agent needs to respond to a signal. I’m currently using Jev, and with the network latency it takes about 0.5-2 seconds for a response, and I’m wondering if I can reduce this latency with a smaller, locally run decision model. I’ve heard of the various local models like Needle 3, but I’m not sure how fast these run on various GPU configurations, i.e. it wouldn’t make sense for me if I need some crazy setup to make it faster than using API. Also curious if anyone has tried running these entirely on CPU.   submitted by   /u/f1yingbanana [link]   [comments]