Some of you might remember deepmoney, the finance models I put on HF a while ago. Since then I've mostly been building LLM pipelines for Chinese A-share news in production. This is the next thing, and I'm open-sourcing it early because I think the data side is more interesting than the model side right now. The problem. Our pipeline had a pretty standard LLM news screener: batch ~100 headlines, ask a big model to score each one for "investment importance", and send the top ones downstream. The explanations it wrote looked great. So I finally checked those scores against what the market actually did in the 30 minutes after each headline was published: - AUC 0.59 in one month and 0.47 in another. That's basically a coin flip, and in one month worse than one. - The "stock relevance" score the model wrote itself had a rank IC of −0.094 against 20-day excess returns. The sign was reversed. Great prose, zero signal. So I threw the scores out and rebuilt the whole thing around one rule: the market is the only judge. What's in the repo - A label factory. For each news item, find the stocks it's about (the "basket") and measure their minute-bar reaction starting at the publish time. That reaction is compared against two things: 500 random baskets drawn at the same minute, and the basket's own normal behaviour. Everything is strictly as-of publish time, no look-ahead anywhere. - Placebo tests for everything. Take the same basket at the same clock time, 5 or 10 trading days earlier. For the "major reaction" label, real items hit 3.2% vs 0.3–0.9% for the placebo. Some label ideas failed this test and got dropped. For example, the market-wide layer came out worse than placebo. - One fun finding. News the old pipeline never extracted at all reacted about as strongly as the news it did pick. So "unextracted = negative" was just wrong. Those rows are now treated as missing labels instead of negatives. - The model design (Jev-inspired). If you saw TypeSafe's Jev ("System One": state + typed questions → calibrated answers in one forward pass, no generation), this is my attempt at that idea with open weights: - Qwen3.8-27B with LoRA. lm_head is ripped out and replaced with ordinal (cumulative-link) heads on several layers, which gives early exit for free. - One prefill gives 5 answers: 30-min price reaction, T+1 reaction, volume burst, how many sources will reprint it, and follow-up coverage. - Numeric market state goes in as soft tokens placed after the text, so the decision position is a "state query" reading the news. - No RL. Jev reportedly needs RL because they train on synthetic data. I have real outcomes, so a proper scoring rule (the ordinal NLL) already gets you calibration at the optimum, followed by temperature scaling after the knowledge cutoff. - There's also a plain LoRA + single-letter-answer baseline (LLaMA-Factory + vLLM logprobs), because the fancy version has to beat the boring one. What's NOT done (being honest here) - The 27B hasn't been trained at full scale yet. Only a smoke test on Qwen3-0.6B so far. There are no model results in the repo, only label validation and baselines. - Baselines to beat on the holdout (after the base model's knowledge cutoff): a text-blind GBM gets around 0.55–0.57 AUC on price reactions and ~0.68–0.70 on volume/propagation. The price part is where reading the text has to earn its keep. - It's a reference implementation on top of my own data stack (ES news index, minute bars, and so on), so rebuilding the labels means plugging in your own sources. You can train straight off the HF dataset though. Links - Code (MIT): https://github.com/zzlgreat/jevm - Dataset, 283k Chinese news items with labels and features, non-commercial research only: https://huggingface.co/datasets/TriadParty/jevm-news The README has the full writeup, including the math on why ordinal heads beat letter-token answers for this kind of thing. Things I'd love opinions on Anyone tried replacing lm_head with task heads on a big decoder and serving it with vLLM? Is prompt_embeds + pooling usable in practice? LoRA vs partial unfreezing for really noisy labels. My bet is LoRA wins, but I'm planning to test it. Better ideas than the ~94% entity linker for Chinese company names? Name collisions are brutal ("农产品" is literally a listed company). Training runs on 8× RTX PRO 6000 soon. I'll post models either way, even if the 27B loses to the GBM 🙃 --- A few things to check before posting: - deepmoney: I only wrote "the finance models I put on HF a while ago" and didn't guess any specific details. If you want to add the model link or what base it was trained on (people on the sub will click through), send it to me and I'll work it in. - The "No RL" bullet says Jev relies on synthetic data. That's from TypeSafe's own public materials, and I wrote "reportedly" so it doesn't read as a factual claim about them. - Title length: it's close to Reddit's 300-character limit. If you want it shorter, you could drop everything from the parentheses onward.   submitted by   /u/Fun_Water2230 [link]   [comments]