I have a feeling that recent benchmarks are unrealistically pushing the bar into areas a normal developer rarely encounters, like binary reverse engineering or one-shotting a complex application, with some strictly chosen design patterns. What we actually want is an accurate model that follows instructions without over-arching it (e.g. without making unnecessary impressive 3D rendering). If I prompt a model to "Generate an SVG of a pelican riding a bicycle," a simple, clean graphic should be a 100% pass. There is no reason for the model to shoehorn in 3D shading, animations, or elaborate backgrounds unless specifically prompted. I think this is one major misconception, judging a model solely on the 3D details it achieves, that was never asked for in first place. So, how many of the added edge-case capabilities in the latest models are actually useful when we daily-drive AI? If they aren't being used, they should be weighted much less on the leaderboards. Instead, what we see, these edge-case capabilities are being rewarded with very high scores. Maybe you don't need a stronger model to achieve exactly the same results in your business area. But I get it, the idea of plateauing the models capabilities is going opposite of the AI ever-growing market valuation, so should be kept pumped forever; just saying.   submitted by   /u/Informal-Trouble2183 [link]   [comments]