An AI response can sound good and still be wrong. An agent can produce a plausible answer without successfully completing the task. I’ve been experimenting with Jev as an alternative to a generative LLM judge. The goal is to make evaluation lightweight enough to run alongside an application, rather than only during occasional offline testing. But what should we actually evaluate? I think it helps to separate three layers: LLM responses: Did the answer address the question and follow the instructions? RAG pipelines: Was the retrieved context relevant? Was the answer supported by that context? Or any other domain specific metric. Agent and tool traces: Was the tool choice appropriate? Does the final answer accurately reflect what the tools returned? The next question is whether we can trust the judge’s results. Structured output alone doesn’t establish correctness. For score-based metrics, a raw 0.8 doesn’t automatically mean humans would accept 80% of similar responses. One approach is to calibrate each metric against representative human judgments: Judge scores + human pass/fail labels → fit per-metric calibration → validate on held-out examples → apply to new scores. This helps ground score interpretation and acceptance thresholds in actual human judgments. It doesn’t automatically improve the judge’s ability to detect mistakes, and the mapping needs rechecking when the task, rubric, or judge changes. What do you evaluate in practice - just the final answer, or retrieval and tool traces too and how do you validate the judge itself? I’m building Typed Evals, an open-source implementation of this approach. For those already running evals: what does your evaluation stack look like, and what have you found actually works (or doesn’t)?   submitted by   /u/Charming_Group_2950 [link]   [comments]