Disclosure: I built this and the repo is mine. One of the two judges, TypeSafe's Jev, is a paid API. The writer, search and memory run locally. This CAN be substituted with any SFT'd classifier or encoder you want to use locally (a finetuned Laya with a different dataset, etc). I was trying to determine if small language models with a retrieval base (MemPalace in my case) and Jev can be used as a homebrewed RAG environment. When Gemma 4 E2B judged its own retrieved passages, its confidence collapsed onto three values, so a threshold on it has almost nothing to move. Jev's score ranked the same passages better. End to end, the pipeline with Jev making every decision did not beat the no-judge baseline: it scored 0.612 against 0.740, missed its main pre-registered bar, made about the same number of mistakes on the questions both answered, and lost because it declined far more questions. Setup: 500 SimpleQA questions, search results frozen on 2026-09-19. Gemma 4 E2B (unsloth/gemma-4-E2B-it-qat-GGUF on llama.cpp) writes the answers. A judge makes every other decision, from whether to search to whether each sentence is backed by its cited passage. Search is self-hosted SearXNG; claude-sonnet-5 grades with the official SimpleQA prompt. Score is +1 right, 0 declined, -1 wrong. Intervals are 95%, bootstrap over questions. On a 20-question pilot I ran it twice over the same passages: Jev as judge, then Gemma judging itself. Gemma's confidence is verbalized: it is asked how sure it is and the number is read back. ## Gemma's confidence sits on three values 3,885 judgments made by both models, on 21 questions: | | Gemma (verbalized) | Jev 1.13.0 | |---|---|---| | Distinct values | 12 | 99 | | Share on its top three values | 99.6% | 34.4% | | Share between 0.10 and 0.90 | 6.3% | 30.3% | | Gate mobility | 11% | 99% | They make the same yes/no call at 0.5 on 82.8% of these. Gate mobility is the share of 0.01 threshold steps from 0.00 to 0.99 that change which answers pass. Noise can game these numbers. Mobility equals the count of distinct values below 1.00 on a 0.01 grid, divided by 100, and uniform random noise scores near 100% on it. The table shows a threshold on Gemma's score can barely move, and nothing about whether moving one helps. That takes a label with no model in it. ## Jev ranks passages better, including where Gemma is certain The label: a passage is positive when the gold answer string appears in its text. Golds of 4 characters or fewer are dropped because they match pages for unrelated reasons. The label is approximate in both directions. Paired on the pilot, 473 passages from 12 questions: - Jev AUROC 0.926 [0.857, 0.977] - Gemma AUROC 0.804 [0.723, 0.852] - Difference +0.122 [+0.046, +0.199] 5 of the 9 questions with answer-bearing passages hold 82 of the 90, so treat those intervals as approximate. Leaving one question out at a time, the difference stays between +0.108 and +0.158. Gemma marked 189 passages 1.00. Only 76 of them (40%) contain the gold string. Jev's score still separates those: AUROC 0.835 [0.662, 0.955], though only 7 of the 12 questions have both kinds of passage in that group. Across all questions with usable gold answers, Jev alone scored AUROC 0.899 [0.881, 0.916] on 9,075 passages from 349 questions. ## Two local fixes I have not tried I did not try logprobs or self-consistency. They are the next arm: - confidence from token logprobs on the verdict (nearly free) - vote fraction over 10 samples (ten generations per judgment) If someone here runs the logprob version and Gemma closes the gap, I will link it from the README. So the claim is narrow: Gemma 4 E2B's verbalized confidence gave a threshold little to move and ranked passages worse than Jev. ## The audit tool `harness/audit.py` audits any judge's confidence scores. It asks whether a threshold can move (distinct values, top-3 share, gate mobility; no labels) and, separately, whether moving it helps (AUROC with a question-clustered bootstrap; needs labels). Random noise passes the first and fails the second. No network or model dependencies. uv run python -m harness.audit my_scores.csv --label label --judge judge --cluster question ## Grader tip: check max_tokens Caught and fixed during development: the grader first ran with an 8-token output cap. The grading model reasons for about 120 tokens before its one-word verdict, so early replies came back empty and were scored unsupported. I raised the cap and made an unreadable reply its own label, left out of the rate. The 500-question grading ran after the fix and reports 0 unreadable replies. ## As a verifier, the judge held up 612 of 623 kept sentences were supported by the passages they cite: 0.982 [0.972, 0.992], against a pre-registered bar of 0.90. That covers only sentences the pipeline kept. ## Open problem: the full pipeline The judged pipeline scored 0.612 [0.562, 0.662]. The no-judge version, which puts the top 8 pages straight into the prompt, scored 0.740 [0.688, 0.792]. Paired difference: -0.128 [-0.178, -0.078]. It cleared three of four pre-registered bars and missed the one that measures whether the judge helps: the gain over baseline. On the 353 questions both answered, it got 23 wrong and the baseline got 22. It declined 142 questions to the baseline's 56; on those 142 the baseline was right 76 times and wrong 15, so the judge gave up about five right answers for every wrong one it avoided. Worth trying: send low scores to a second search or a stronger model before declining, and set the decline rate from the baseline's error rate. ## Caveats - The Gemma comparison is small: 12 questions for AUROC, 21 for spread, passages correlated within each question. - The AUROC label is a string match, and the grader is a model. - The AUROC numbers come from raw judge records that stay local. The script (scripts/informativeness.py) and its output are published. Repo: https://github.com/clduab11/jev-test Statistics: https://huggingface.co/datasets/clduab11/jev-calibration-statistics I'm not sure if these observations will help anyone as far as their own locally-deployed Jev-like classifiers, but hopefully this gives good information for those in the space!   submitted by   /u/clduab11 [link]   [comments]