Follow-up to my CQ post from last week. That post covered the 4-bit base model; this one is about the adapter. The LoRA vs GCMP runs on 4-bit bases have now finished. GCMP is a LoRA replacement we built. On Qwen3.8-27B it trains 2.9M parameters (~11 MB), vs ~159M (~320 MB) for LoRA r=32 on the same 7 projections. That's 55× smaller. Accuracy: same as LoRA All adapters were trained on Alpaca for 500 steps with identical settings (batch 2×4, length 512, LR 2e-5, seed 42). Every row was scored with the same lm-evaluation-harness code and questions, greedy decoding, thinking off. Comparisons use paired McNemar tests. Base + adapter Params Answer loss GSM8K SQuAD BoolQ ARC-C MMLU IFEval TruthfulQA BF16, no adapter – 1.678 0.683 0.746 0.874 0.588 0.839 0.805 0.542 BF16 + LoRA r=32 159M 1.480 0.903 0.776 0.896 0.612 0.844 0.825 0.518 BF16 + GCMP 2.9M 1.471 0.890 0.760 0.892 0.618 0.846 0.815 0.514 NF4 + LoRA r=32 159M 1.464 0.793 0.748 0.880 0.608 0.840 0.835 0.528 NF4 + GCMP 2.9M 1.458 0.833 0.756 0.878 0.618 0.837 0.840 0.522 BF16 base (no quantization at all): no significant difference on any benchmark (all p ≥ 0.2). NF4 base: GCMP is +4.0 on GSM8K (p=0.04) and tied on everything else. Loss: GCMP's answer loss is slightly lower on both bases, and its validation loss was lower at all 10 checkpoints on both. One significant win on one task isn't enough to call it better, so the claim is same quality as LoRA at 1/55th the size. Why the size matters For one fine-tune, 320 MB vs 11 MB doesn't matter much. It starts to matter when you have many: Multi-tenant serving: one fine-tune per customer or task on a single base model. In vLLM, GCMP adapters load once at startup and each request picks its adapter by model name. A batch can mix adapters and runs in one kernel call per layer, with no merging or swapping. Resident GPU memory: vLLM keeps each adapter in its compact form and never builds a dense weight update. Memory is about 11 MB per adapter, plus a fixed ~1 MB routing buffer and a shared workspace of ≤ ~64 MB that doesn't grow with adapter count. Storage and loading: 1,000 adapters are ~11 GB instead of ~320 GB. That's faster to ship, cache and hot-load. Edge and small GPUs: an extra fine-tune costs megabytes, not hundreds of MB. Estimated 1 adapter 100 1,000 LoRA r=32 320 MB 32 GB 320 GB GCMP 11 MB 1.1 GB 11 GB These are estimates from adapter size. I haven't measured memory and throughput at 100 or 1,000 adapters yet, and that's the next benchmark. What's not good yet Training speed: GCMP takes ~2× LoRA's step time (BF16 on Unsloth: 2.74 vs 1.37 s/step; NF4: 5.6 vs 3.2). It does use slightly less training memory (21.5 vs 23.8 GiB on NF4). Baseline: r=32 is a generous LoRA baseline. r=4/8/16 comparisons are next, and I'll post them either way. Serving: vLLM serving of GCMP currently needs a CQ base. On full-precision models it runs in Transformers for now. Scope: one model, one dataset, one run per case. The quick benchmark subset (200–1,140 questions per task) detects ~5-point differences, not 1-point ones. Quick update on CQ The CQ base still matches BF16 (no significant difference on any of the 9 benchmarks) while NF4 loses 12 points on GSM8K. After fine-tuning, though, the gap closes: CQ + GCMP vs NF4 + GCMP: tied on GSM8K (83.0 vs 83.3). Both 4-bit stacks vs BF16 + GCMP: both end up ~6 points behind on GSM8K. So CQ's GSM8K lead didn't survive fine-tuning in this test. CQ + LoRA is being retrained, because the adapter from our older trainer turned out to be broken, and I'll add it when it's done. Code and licensing The Python code is open source. The CUDA kernels are compiled with a license check, which is free during early access. Repo, with the commands to reproduce every number: https://github.com/thousandml/training-factory Write-up with the full tables and the serving memory breakdown: https://huggingface.co/blog/thousandml/cq-gcmp-4bit-finetuning If you run multi-tenant fine-tunes, I'd like to know: is adapter memory or count actually a bottleneck for you, or is it something else?   submitted by   /u/textclf [link]   [comments]