Hello All! I built a platform to fine-tune open models. I am focusing on fine-tuning 4-bit quantized models by QLoRA instead of full precision fine-tuning to reduce GPU costs. The quant that this finetuning uses is called CQ (Coded Quantization) which is a calibration-free quantization that I built that performs as good as leading quants ensuring that the fine-tuning is using a good base. For example, I compared KLD of my CQ 4-bit with unsloth-bnb-4bit (Unsloth have other quants but this is the one they use for QLoRA) on Qwen 3.8 27B using the Wikitext-2 dataset and got this: Quant Disk Size (GB) Mean KLD Median KLD 99% KLD Top 1% Agreement CQ 4-bit 18 0.02823666 0.01282929 0.97606068 92.419% unsloth-bnb-4bit 22.4 0.03297115 0.01437706 1.18053893 91.954% So I have a better quantized base to start from. I ran QLoRA finetuning with Training Factory of thousandml/Qwen3.8-27B-CQ-4bit on the Alpaca dataset using an A40 (48GB GPU) and here is the performance measures: ===== CQ TRAINING BENCHMARK ===== { "optimizer_steps": 100, "training_wall_seconds": 1557.1155763969873, "warmup_steps_excluded": 10, "median_steady_step_seconds": 14.990616369002964, "steady_optimizer_steps_per_second": 0.0643880924733763, "peak_pytorch_allocated_gib": 25.532413005828857, "peak_pytorch_reserved_gib": 33.005859375, "notes": [ "Step timings include gradient accumulation and Trainer overhead.", "PyTorch peak excludes allocations outside its CUDA caching allocator.", "Token counts from the collator may include prefetched batches; do not interpret as exact consumed tokens.", "Wall time includes any initialization inside Trainer.train and callback overhead." ], "collated_input_tokens_including_prefetch": 130236, "collated_nonpadding_tokens_including_prefetch": 106290 } So great QLoRA finetuning with reasonable GPU performance. Get Started with Training Factory Here are simple steps to use: Install: ​ sudo apt-get update sudo apt-get install -y git-lfs git lfs install git clone https://github.com/thousandml/training-factory.git cd training-factory chmod +x install.sh ./install.sh 2. Download a CQ 4-bit base: hf download thousandml/Qwen3.8-27B-CQ-4bit --local-dir ./models/Qwen3.8-27B-CQ-4bit 3. Train a LoRA adapter: python training/train.py --model ./models/Qwen3.8-27B-CQ-4bit --data training/examples/train.jsonl --output ./outputs/my-lora --targets q_proj,v_proj --kernel-module cq.cq_kernels --text-only --max-steps 2 --max-length 128 --batch-size 1 --grad-accum 1 4. Serve base + adapter: vllm serve ./models/Qwen3.8-27B-CQ-4bit --quantization cq --enable-lora --max-lora-rank 16 --lora-modules my-lora=./outputs/my-lora 5. Call the fine-tuned adapter: # After the server is ready: # Check that the adapter is listed as "my-lora" curl http://localhost:8000/v1/models # Request a response from the fine-tuned model, not just the base checkpoint curl http://localhost:8000/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "model": "my-lora", "messages": [ {"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Explain how LoRA fine-tuning works in one sentence."} ], "max_tokens": 128, "temperature": 0.2 }' I have the following 4-bit quantized models to be used as base (will add more soon) for now: thousandml/Qwen3.8-27B-CQ-4bit thousandml/Llama-3.1-8B-Instruct-CQ-4bit thousandml/Llama-3.3-70B-Instruct-CQ-4bit thousandml/Qwen3-Coder-Next-CQ-4bit Please give it a try and let me know if it works with you. Your feedback is greatly appreciated!   submitted by   /u/textclf [link]   [comments]