Hi all! After all the jev hype I wanted to try by myself and got qwen3.8 27b to mix-and-match open implementations on top of ninfer, and to be honest I'm quite impressed with the results, latency is not great but accuracy is almost on par, this is a very dirty POC but it kinda works (there's still some bugs and incorrect allocation, still need to figure out how to properly integrate into ninfer). Source code at: https://github.com/knoopx/ninfer Benchmark Measured against JevBench v1.2, a decision-model benchmark. The public set is 231 typed decisions (48 easy, 111 hard, 72 original): each is a state plus a bounded rubric, answered with a typed result and per-option probabilities. The run drives the native POST /v1/decisions route directly (state + question in, typed probabilities out), not the chat-completions verbalization path. Setup: model https://huggingface.co/knoopx/Swift-Qwen3.8-27B-NInfer , RTX 5090, three in-flight decisions. All 231 tasks scored, 0 errors, 0 admission blocks. Wall clock 19 s. Headline Metric Value Accuracy (231 public) 84.4% (195/231) ECE (10 bins) 0.045 Brier 0.081 Latency p50 / p95 127 ms / 699 ms Speed axis (JevBench ×2 + 0.15 s) ~81.7 Accuracy by tier and type Tier choice noul score total easy 100% (36/36) 100% (12/12) — 100% (48/48) original 97% (35/36) 95% (23/24) 100% (12/12) 97.2% (70/72) hard 70% (47/67) 73% (28/38) 33% (2/6) 69.4% (77/111) all 84.9% (118/139) 85.1% (63/74) 77.8% (14/18) 84.4% (195/231) Hard tier by family Family Acc temporal-numeric 33% (5/15) probability 50% (5/10) tradeoff 67% (4/6) long-policy 68% (13/19) multi-hop 72% (13/18) judge-hard 76% (13/17) ambiguous 86% (6/7) trap 88% (7/8) adversarial 100% (6/6) routing-hard 100% (5/5) Latency (seconds) Scope p50 p90 p95 p99 max mean all 0.127 0.619 0.699 0.896 0.925 0.238 easy 0.100 0.103 0.453 0.691 0.736 0.134 original 0.101 0.123 0.123 0.309 0.331 0.109 hard 0.286 0.694 0.774 0.911 0.925 0.366 Hard-tier p95 (774 ms) is prefill-dominated by the ~3.7k-token states. Tail latency (p95 and above) is queueing-sensitive: a single decision worker serializes execution, so a task can queue behind heavier in-flight decisions, inflating per-tier tails (visible in the easy row). p50 and mean reflect the compute cost; accuracy and calibration are unaffected by queueing. Calibration 36 of 231 wrong. ECE 0.045 (10 bins), Brier 0.081 — the model is well calibrated overall. The five most-confident wrong answers: Task Conf Predicted Gold hard-opus-b-tradeoff-01 0.949 escalate_to_security escalate_to_engineering hard-opus-a-probability-08 0.852 yes no hard-sol-b-judge-hard-02 0.852 yes no hard-opus-c-temporal-numeric-02 0.798 no yes hard-opus-a-probability-04 0.729 late on_time Field comparison (same 231 public tasks) Every published system's public-set accuracy, against its published JevBench composite (the composite is a geometric mean over Intelligence, Calibration, Speed, and Cost on the full 534-task set, so it is not directly comparable to a public-set accuracy). This run has no published composite (self-hosted, no Speed/Cost measurement): Rank System Score Pub acc 1 Jev 1.13.0 (TypeSafe AI) 75.4 86.6% 2 SemIf, formerly OpenJev (Qwen3.5-4B) 74.7 81.0% 3 djev (Maisa, diffusion-gemma) 74.3 84.0% 4 openJev Verdict 1.4 72.5 57.6% 5 Laya (ModernBERT-large 421M) 70.1 58.4% 6 open-alternative-jev (Qwen3.5-4B) 69.8 74.0% 7 system-one-open (Gemma 4 E2B LoRA) 68.9 73.2% * NInfer ukisai/Swift-Qwen3.8-27B (this run) 68.4 84.4% 8 OpenJev (DiffusionGemma 26B NVFP4) 67.7 81.8% 9 SimpleJev Qwen3.8-27B 67.3 86.6% 10 jeff (GLiFormer 400M) 66.9 62.8% 11 kev 0.6B 66.7 66.7% 12 openjev-sglang (Qwen3.6-35B-A3B) 66.3 85.3% 13 openJev Verdict (ModernBERT-base 151M) 66.2 55.4% 14 GPT-5.6 Luna (low reasoning) 66.2 97.4% 15 open-jev-deberta-v3-large (CPU) 64.6 52.4% 16 SimpleJev Qwen3.6-35B-A3B 63.8 81.4% 17 Bespoke Nimble 9B 63.7 67.5% 18 kev 0.5B 63.1 49.4% 19 kev 4B 62.2 66.2% 20 Gemini 3.1 Flash-Lite 60.9 87.0% 21 kev 8B 58.3 71.4% 22 DeepSeek V4.1 Flash (thinking) 57.8 97.8% 23 system-one (Qwen3-8B) 56.6 71.9% 24 GLiNER2 (Fastino) 53.0 58.0% — classifier.dev (fast tier) 84.8 85.3% — Qwen3.8 27B (Chutes TEE) 25.5 71.9% — Needle 3 (options as tools) 19.2 22.1% — Needle 3 (Cactus, 2-bit, CPU) 16.8 22.5%   submitted by   /u/Unlucky-Message8866 [link]   [comments]