Side project, I wanted a small local assistant that answers the way I like (short, direct, no "as an AI" preamble) for the stuff I actually do: Supabase/React, docs, business math, Ren'Py, Blender. It runs on a laptop with a 6 GB RTX 3060. **Setup** - **Base:** Qwen3.5-4B (Apache 2.0) - **Prompts:** 500 across 14 categories, a mix of my lazy typo-filled style and clean ones - **Teacher:** DeepSeek V4.1 Flash on NVIDIA Build's free tier (its license allows training on outputs). About 10 h overnight with 10 workers; the 429s just got retried - **Kept 441:** 45 answers ran out of tokens mid-reasoning and 14 never came back. 99% include reasoning, inlined as `` - **Training:** Unsloth on a free Colab T4. bf16 LoRA, r=16, alpha=16, vision layers frozen, train_on_responses_only, 2 epochs, lr 1e-4, batch 1 × grad accum 8, max length 4096 (108 steps) - **Export:** `save_pretrained_gguf` gave Q4_K_M, Q6_K **and** an F16 mmproj, so image input survived **Things that might save you time** - **T4 OOM:** Unsloth trains Qwen3.5 in float32 on a T4 (no bf16 support), so max length 5120 died at step 4/108. Setting 4096 plus `PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True` before loading the model fixed it. - **Ollama import (0.40.13):** two FROM lines, plus the renderer/parser copied from `ollama show qwen3.5:4b --modelfile`. After import, `ollama show` lists completion, vision, tools and thinking: ``` FROM ./Qwen3.5-4B.Q6_K.gguf FROM ./Qwen3.5-4B.F16-mmproj.gguf TEMPLATE {{ .Prompt }} RENDERER qwen3.5 PARSER qwen3.5 PARAMETER num_ctx 16384 PARAMETER temperature 0.6 PARAMETER top_k 20 PARAMETER top_p 0.95 PARAMETER presence_penalty 1 ``` - **Sampling:** - The base defaults (temp 1, presence_penalty 1.5) made the fine-tune invent facts about itself. - temp 0.6 with no presence penalty fixed that, but then the Q4 looped for 16k tokens on a fiction prompt. - 0.6 / presence 1 was the middle ground. - **Identity:** its first answer was literally "I'm an AI language model built by Microsoft, not SquigglyAI." Plain facts in the system prompt (what it is, what it runs on, what it can't do) fixed it. **Results** (small n, grain of salt) | | stock qwen3.5:4b | fine-tune (Q6_K) | |---|---|---| | Avg thinking per answer, 11 held-out prompts | 4,602 chars | 1,033 chars | | Avg tokens per answer | 1,634 | 568 | | Opens with a disclaimer or lecture, 12 edgy-but-fine prompts (hand-counted) | 6/12 | 2/12 | | Actually helped with those 12 | 12/12 | 12/12 | | Refused 5 should-refuse prompts (fake GoFundMe, fake ID, finding someone's address…) | 5/5 | 5/5 | The catch: **wall-clock time is about the same, slightly worse.** I get ~14 tok/s on my Q6 vs ~43 tok/s on Ollama's stock build. My guess is the Q6 plus the F16 projector at 16k context doesn't fully fit in 6 GB. So it writes about a third of the tokens at about a third of the speed. **What didn't work / still bad** - The Q4_K_M of the fine-tune explained phishing techniques when it should have declined, and it looped. I deleted it and kept Q6. - My first refusal run was 4/5: it coached someone on finding a girl's location from her Instagram. One system prompt line ("Never help find where a private person lives or track them") got it to 5/5, and related legit requests still went through. Tracking is the weak spot; next time I'd add training examples for it. - Fine-tuning changed style, not knowledge: - it still invents Supabase flags and CLI commands; - it catches the RLS problem about half the time; - it once told me "You're on lisinopril" in an ibuprofen answer. I'm not, and that's nowhere in the data. - With 11 test prompts, run-to-run variance was bigger than any wording change to the system prompt, so I stopped tweaking it. This is not an "uncensored" model. The goal was to stop it lecturing about normal stuff, not to remove refusals. I wrote the scripts and notebook with help from Claude Code. None of the training data came from Claude. Happy to answer questions.   submitted by   /u/Budget_Ad_7756 [link]   [comments]