We’ve been testing “System One” models, also called decision models. Instead of generating text token by token, they return a typed decision with a probability for each option in a single pass. To compare them, we put together the AIM System One Index. It combines three benchmarks: Browser tasks: 50 tasks, one attempt each. Classification: 1,655 single-decision questions, such as security advisory severity, FDA recall class, paper topic and checking whether a browser task is done. Bias: multiple-choice questions, with the “cannot be determined” option rotated through every position. Each score is converted to a z-score, so 50 is an average model and every 10 points is one standard deviation. Cost and latency are shown next to the index but don’t count toward it. Across 57 models, three are tied at the top with 60: AutoJev-27B, d1-ultra and decider-12b. Newcomer d1-ultra has the best browser score at 74% and costs just $0.026 per 1,000 decisions. GLiDE also does well on browser tasks at 72%, but it takes about 15.7 seconds per decision. Laya is still the fastest at 30 ms per decision. Decider Flash V1.2 is now the cheapest, at $0.007 per 1,000 decisions. Full leaderboard and methodology: https://aimultiple.com/system-one. We’d love feedback on which models or tasks we should add next.   submitted by   /u/AIMultiple [link]   [comments]