Running the app I wanted a terminal helper that works fully offline, so I started building dale. You type dale "show the 5 largest files in this folder", a small fine-tuned model writes the command, and a deterministic risk checker labels it safe, caution or danger before anything runs. Nothing runs without you confirming it. Scores on 471 held-out requests in English and Spanish, every model in Q4_K_M on the same llama.cpp: Model GGUF Correct Accuracy 95% CI dale-9b (GPU) 5.6 GB 420 89.2% 86.0–91.7 dale-4b (GPU) 2.7 GB 410 87.0% 83.7–89.8 dale-2b 1.3 GB 385 81.7% 78.0–85.0 dale-0.8b (CPU) 529 MB 374 79.4% 75.5–82.8 Qwen3.5-9B, stock 5.7 GB 296 62.8% 58.4–67.1 Gemma 4 E4B-it, stock 5.0 GB 296 62.8% 58.4–67.1 Qwen3.5-4B, stock 2.7 GB 257 54.6% 50.0–59.0 Gemma 4 E2B-it, stock 3.1 GB 254 53.9% 49.4–58.4 Qwen3.5-2B, stock 1.3 GB 152 32.3% 28.2–36.6 Qwen3.5-0.8B, stock 533 MB 70 14.9% 11.9–18.4 On a 2020 laptop with no GPU, the 0.8B answers in 1.7 s (median). The 4B does 1.2 s on a 16 GB GPU through Vulkan. It's all one 51 MB binary with llama.cpp inside, no Python and no server. Comparison finetuned family What the difference looks like. Real answers from the test (requests trimmed), dale-0.8b vs stock Qwen3.5-9B: Request dale-0.8b Stock Qwen3.5-9B Temporarily freeze PID 22601 without terminating it. kill -STOP 22601 Writes to /proc/22601/oom_score_adj. Freezes nothing Copy offer.pdf into /mnt/usb/, but leave any existing offer.pdf there untouched. cp -n …/offer.pdf /mnt/usb/ sudo cp …/offer.pdf /mnt/usb/, which overwrites it Can you give me the prime factorization of 1234567890 from the terminal? factor 1234567890 A 180-character python3 -c one-liner that's a syntax error I just need the total disk usage of /opt/model-cache, in a readable size. du -sh /opt/model-cache os.path.getsize() on the folder: the size of the directory entry, not its contents Lock the password of the local account archivebot. sudo passwd -l archivebot sudo passwd archivebot, which sets a new one For this Bash session, make ll an alias for ls -alF. alias ll='ls -alF' Refuses: "not possible with a single shell command" Cherry-picked, obviously: there are 120 cases only dale gets right and 42 only the 9B gets right. The stock 9B also likes sudo: it added it to 35 commands that didn't need it (dale: 5). Comparison failures The thing I wish someone had told me earlier: after four rounds of data, the 0.8B got stuck at 74%. I tried DPO on its own failures, a fine-tuned 27B teacher with five distillation setups, a third epoch, lr sweeps. Nothing moved the test. I f**king lost several days into this :(, but at least I learnt what is important in the process. Then I looked at how the requests were written. My training requests were short and clean. The test ones were longer and stacked constraints, like "only names, sorted by date, no subfolders". https://preview.redd.it/oocu4a0qoe0i1.png?width=2304&format=png&auto=webp&s=fa30e71800cbc722c89f76a9210744f6acc7fb49 I generated new requests matching those stats, added about 9k to the set and retrained. 348 → 375 correct (p = 0.001). https://preview.redd.it/11bdtj5soe0i1.png?width=2304&format=png&auto=webp&s=db8e2a2158263f25100424512595b7d2031a1025 So if a small model plateaus on a narrow task, look at how your data is written before you reach for a fancier method. Caveats: the baselines are untuned and get an 8-shot prompt, so this is "tuned small vs prompted big", not a ranking of models. The test is synthetic and judged by two LLMs from different providers (TOP TIER). All of this work I did on rented GPU's (RTX 5090 and RTX Pro 6000). Code is MIT, the models are Apache-2.0 GGUFs that work with plain llama.cpp. Write-up with every chart and the dead ends: https://manuelgs.com/en/blog/dale-case-study/ Code: https://github.com/manuelgscom/dale Models (0.8B to 9B): https://huggingface.co/collections/manuelgscom/dale Has anyone actually made distillation from a much bigger teacher pay off below 1B? I couldn't. Any advice would be well welcomed :)   submitted by   /u/AlternativeEnough352 [link]   [comments]