Everything in the title, Is jev, text embeddings but without chunks? and a confidence score? Without a ranking or reranking (that part I haven't tested yet)? FYI for those who didn't follow. It's api has 3 modes: Choice – pick one option — up to 255 classes Score – rate on 2–10 ordered levels Noul – yes/no, returns probability (0–1) — 2 classes Even in Choice mode it output: choice – the winning option probabilities – distribution over all options confidence – 0–1, high when one option dominates That confidence is supposed to be trustworthy compared to measuring logits with traditional LLM BTW I ran some benchmarks on a synthetic dataset I made with Claude here are the best results: Model ms per doc Doc class: accuracy / macro-F1 / calibration error Language Donor chart Is clinical Urgent decider-0.8b 288 0.896 / 0.897 / 0.039 1.000 0.997 0.843 0.902 kev-0.8b 272 0.826 / 0.798 / 0.270 0.737 0.908 0.768 0.692 litjev + Qwen3.5-0.8B 553 0.451 / 0.395 / 0.063 0.958 0.437 0.754 0.118 laya typed-decisions 119 0.308 / 0.309 / 0.079 0.549 0.821 0.748 0.599 laya root 64 0.294 / 0.287 / 0.469 0.636 0.765 0.745 0.541 laya multilingual 59 0.255 / 0.293 / 0.499 0.356 0.574 0.641 0.499 Here for the confidence index of decider 0.8B on a 357-document dataset. Stated confidence n Actual accuracy < 0.50 18 0.39 0.50–0.70 41 0.56 0.70–0.80 21 0.81 0.80–0.90 50 0.94 0.90–0.95 31 0.97 0.95–0.99 82 1.00 ≥ 0.99 114 1.00 It tracks reality closely and is if anything slightly under-confident at the top. Its 37 errors have a median confidence of 0.59; only one was above 0.90 (0.92). That makes a human-in-the-loop threshold work well: Accept if confidence ≥ Auto-classified Accuracy on those Errors let through 0.70 83.5% 0.973 8 0.80 77.6% 0.986 4 0.90 63.6% 0.996 1 0.95 54.9% 1.000 0 KEEP IN MIND this dataset was probably made with the same model that made the training set for decider 0.8B?   submitted by   /u/No_Afternoon_4260 [link]   [comments]