Okay so there are strongish open jevlikes. But jev's official benchmarks seem to indicate they think open models have bad calibration, and their ranking of which they think performs best doesn't match the open benchmarks. Which could mean the open benchmarks suck, or could mean jev's official benchmarks suck. But either way, it would be cool if there was an open benchmark that at least correlated strongly with the official one, because that would be a hint that maybe it's measuring something more like what the official one does. The official one: official one official one (intelligence) What I'd like to see is a local model which matches them on their benchmarks; but failing that, I want to see what sort of data one has to put in a benchmark in order to get these results. because they're not the results we see on the jev decision index. snapshot of the index results as I post this: jev decision index results so... that's odd. now obviously the official benchmarks don't particularly test that many others, but kev coming in second on the official one but not even on the first page on the unofficial one seems like a hint something is wrong! Are there other good jev benchmarks? the only other I currently know of is jevbench, which is kind of a ghost town in terms of results count (and also has a website with a very bad ratio of content to information, and only tests on game playing strength.) Ideally I'd like to see a benchmark which has breakdowns for individual skills tested, eg image parsing, video parsing, math, code next step, game playing, etc, and ideally that benchmark would cover different ways to present the same information to each model in case the models disagree on favored input format for different types of information. If I have to I might make my own, but if I do I probably won't post it, because I just can't be bothered. Someone who can be bothered would get my appreciation!   submitted by   /u/lahwran_ [link]   [comments]