I've been experimenting with tiny models for WebBrain, my open-source browser agent. The long-term goal is to run as much intelligence as possible locally, ideally through WebGPU. One approach I've been particularly interested in is Prism ML's Ternary Bonsai: compressing a 9B model to roughly 2 GB using ternary weights (-1, 0, +1). Prism released the weights and inference runtime, but not the complete BF16-to-ternary compression pipeline. You can fine-tune their existing ternary models, but there's no publicly available way to compress your own BF16 fine-tune into their format. So I tried building one. Introducing OpenBonsai, an Apache-2.0 framework for post-training ternary quantization, compatible with Prism's PTQ1_0 GGUF format. How it works A recent Intel Labs China paper, "Attend to Your Own Thoughts" (Wang et al.), inspired part of the approach: using the model's own reasoning traces for calibration rather than relying exclusively on generic text. The pipeline combines: Signed Hadamard rotations to reduce weight outliers Sequential GPTQ to quantize weights to {-1, 0, +1} FP16 group scales (one per 128 weights) Self-generated reasoning traces for calibration Scale distillation against the original BF16 model's output distributions Export to Prism's PTQ1_0 GGUF format Results Qwen3.5-9B: ~18 GB → 1.997 GB That's the complete GGUF file, not just the transformer weights. It includes the compressed embeddings, LM head, all other model tensors, group scales, tokenizer, chat template, and metadata. Runtime memory requirements are higher. The compression runs in approximately 5 hours on a single RTX 5090 (32 GB). Our current compressed model scores: Benchmark Original BF16 OpenBonsai Prism Bonsai 2 GSM8K (thinking off) 77/100 65/100 77/100 HumanEval (thinking off) 44/50 34/50 45/50 GSM8K (thinking on) 36/50 32/50 46/50 HumanEval (thinking on) 22/25 14/25 24/25 Some interesting findings 1. Distillation helps substantially. Scale distillation improved development-set NLL from 3.48 to 2.89 and ARC-Easy accuracy from 13/20 to 19/20. 2. Calibration data matters enormously. Switching to self-generated math and code traces improved reasoning and coding performance. Adding more code-focused calibration data increased HumanEval from 19/50 to 34/50 in our initial evaluation. 3. The approach generalizes beyond Qwen3.5. We also compressed Qwen3-8B into a 1.8 GB GGUF using the same pipeline, without changing the generic dense adapter. It runs in Prism's llama.cpp fork. 4. Fully ternary endpoints are feasible. We quantized the embedding and LM head too, rather than leaving large FP16 tensors in the final file. Limitations This is still a research preview. Prism's official ternary models outperform ours on most reasoning and coding evaluations. Some code-training prompts paraphrase HumanEval tasks, so our initial 34/50 result is likely optimistic. We document the overlap and additional held-out evaluation in the repo. The tested configurations require a 32 GB GPU for compression. Lower-memory configurations haven't been validated. Inference currently requires Prism's llama.cpp fork. Upstream llama.cpp and WebGPU aren't supported yet. Architecture support is still limited. I've included the failed experiments, ablations, evaluation scripts, and intermediate findings. The idea is to make the compression process reproducible and improve it openly. Links Framework (Apache-2.0): https://github.com/esokullu/openbonsai Compressed Qwen3.5-9B model: https://huggingface.co/webbrain-one/openbonsai-qwen-3.5-9b Project website: https://openbonsai.org Intel Labs paper: https://arxiv.org/abs/2608.01078 I'd be particularly interested in feedback from anyone working on ternary quantization, GPTQ, or low-bit inference. There is clearly still a quality gap between our approach and Prism's, but I think making the BF16-to-ternary pipeline open and reproducible is a useful starting point. And if anyone wants to try compressing their own fine-tune, I'd love to hear how it goes.   submitted by   /u/ButtercupLyn100 [link]   [comments]