I know this is a controversial take, but I would like to hear others’ opinions: LLMs have become much more useful, but I’m not convinced their “raw intelligence” has improved nearly as much as their overall performance suggests. Context: I work with LLMs as a SWE, as a hobbyist experimenting with local models, and professionally integrating AI into products. That includes building our own harnesses: context retrieval, prompts, and custom tools that give models structured access to complex data. By “raw intelligence,” I mean using what is given in the most efficient way to solve a problem, especially when information, tools, and time are limited. Given plenty of time and tools, planning and long-term agentic capabilities become more important—but that’s a different comparison. Three observations brought me here. First, building production harnesses: once a harness+model worked well, achiving a given score over a golden dataset, switching to a newer frontier model rarely improved results. Most failures within our time/precision constraints came down to context engineering or inadequate tools. These tools included code generation for analysis and complex relational queries, so plenty of "time to shine" for smart models. Newer models sometimes found more creative workarounds, but these were usually too slow or unreliable for production. We ended up improving the harness anyway, leaving the newer models' gains less important. Second, software engineering: newer models have more often improved results here, but the same model can also perform quite differently across harnesses. In complex codebases, that shows up in how much correction and review senior engineers have to provide. The harness seems to do a lot of heavy lifting here, too. Third, local models: smaller models have started performing so well that I question how many parameters efficient problem solving actually needs. Frontier models clearly add broader knowledge, nuance, best practices, and specific baked-in skills. But I don’t always see a proportionate advantage in using it to solve a problem, when the relevant knowledge is already provided. I’m not saying models haven’t gotten smarter. Better tool calling, fewer hallucinations, and longer context windows are real improvements that make them much more useful. Nor do these observations prove that intelligence needs few parameters or that frontier progress has stalled. But my broader take is: efficient problem solving may need fewer parameters than generally assumed, the frontier has moved less in “raw intelligence” than the overall gains - and talk of approaching superintelligence - suggest, and much of the progress over the last 12–24 months comes from cumulative model improvements and better harnesses, rather than a dramatic or exponential jump in raw intelligence. This take seems at odds with both the hype and the doom we see in the media, which I suspect are co-fueled by the upcoming AI IPOs. Does it match others’ experience - or are you seeing vast reasoning gains in frontier models that better context and tools can’t make up for?   submitted by   /u/Reasonable_Goat [link]   [comments]