There’s a lot of hype around JEV right now, as well as local/open versions like KEV and Laya. However, I think there’s one use case that could actually benefit from decision models (or, let’s be honest, one-shot classifiers lol): evaluations for agents. Right now, LLM-as-a-judge evaluations are super expensive (both in cost and latency). What I’m excited about: Running evals locally. Sending a request to evaluate your system does not make sense from a cost or latency standpoint. At millions or billions of decisions, generating text for every judgment gets expensive. I remember trying to speed up Ragas using OpenAI’s parallelism recipes and still running into bottlenecks. That overhead matters even more inside an agent loop. Generalization. A classifier you can adapt through natural-language questions, without starting a new fine-tuning project for every task, could make many more use cases practical. The test is how well it works on your own data. Picking the right model for the job. Classification and regression never stopped being useful. We got so excited about text generation that we started applying it to problems whose answers are a label, a probability, or a score. I'd love to get your feedback, github link here: https://github.com/openlayer-ai/jevals   submitted by   /u/byebaybay [link]   [comments]