Disclosure: I built this model. I'm posting because I think the approach is more interesting than the model itself, and I'd like feedback from people who run things locally. The idea. A lot of "LLM in production" work in finance comes down to classification. Which event is this headline about? Is this Fed sentence hawkish? Does this 10-K passage flag going-concern doubt? People usually prompt a model, sample tokens and parse the output. That's slow, the cost per call varies, and the confidence numbers mean nothing. Arbiter-8B drops generation completely: Base: Qwen3-8B, with a decision head over options, trained jointly with a rank-32 LoRA Output: a calibrated probability for every option from one forward pass. No sampling and no parsing. Question types: pick one of up to 16 options, yes/no, or an ordered score. Each option has a short definition, so you can ask the same text different questions. Distillation: a 27B teacher labelled about 120k decisions on news and 10-K passages. The student trained on the teacher's full soft distributions, not argmax, plus a human-reviewed 10-K set. Option order is shuffled on every presentation to kill position shortcuts. This cut FOMC answers that flip under reshuffling from 15.5% to 4.8%. Calibration: one temperature per question type, shipped with the weights. EDT expected calibration error is 0.012. Results (held-out data): Benchmark Arbiter v0.5 OpenAI Decisions API Jev 1.14 Qwen3-8B base EDT event detection (1,933 articles) 94.2% 90.8% 88.3% 89.3% FOMC hawk/neutral/dove (496 sentences) 67.5% 67.3% 66.7% 55.0% SEntFiN accuracy (1k headlines, unseen) 75.7% 78.0% 81.6% 75.4% The EDT gaps are significant (McNemar, z ≥ 7.5). On FOMC the top three are a statistical tie, but it's +12.5 over its own base. SEntFiN is where it loses. It has the highest signal recall (92.0%) but the lowest precision (72.2%), so it over-calls positive and negative where the dataset labels neutral. Part of that is deliberate (it reads investor implication rather than surface tone), but I won't pretend it's a win. Running it locally: GGUF builds for llama.cpp, Ollama and LM Studio stay within 1.2 points of full precision on every benchmark. hf download ternary/Arbiter-8B --local-dir Arbiter-8B Weights: https://huggingface.co/ternary/Arbiter-8B GGUF: https://huggingface.co/ternary/Arbiter-8B-GGUF Write-up: https://ternary.capital/blog/arbiter-8b/ Personal Blog: https://jdsemrau.substack.com/p/how-good-is-arbiter-8b-really Questions for the sub: has anyone else put classification heads on small LLMs instead of using constrained generation? And has anyone found a clean way to expose per-option logprobs through Ollama for this kind of thing?   submitted by   /u/d3the_h3ll0w [link]   [comments]