Hi Guys!, Recently I was making a research and development, but I do not have time to go further with this idea (kids, work, etc.). But maybe it will be interesting for someone. Below is formatted overview of the project generated by Claude. I do not think it would be something big, but maybe interesting approach that can be used in some of the cases. I believe that it is not AI slop, you can use those information's and try by yourself using Claude or Astra. The idea was my own. Not sure if outdated or not, it can work also with MTP. Please, just don’t be unkind. I’m sharing my idea with you, and anyone is welcome to experiment with it. TL;DR: If a model's Q8 doesn't fit in your VRAM, keep only a 2-bit copy on the GPU and the residual (W − Ŵ, 6–8 bit) in system RAM. The 2-bit copy proposes a tree of candidate tokens. The verifier is the 2-bit copy plus the residual: Ŵx on the GPU and Rx on the CPU, with only activations crossing PCIe. It re-scores the whole tree in one batched pass. In my experiments the verifier is more precise than Q8_0 (about half the KL vs bf16). The 2-bit draft that reads the verifier's exact KV cache beats Google's official Gemma 4 drafter by 25–30% in tokens accepted per check. A CPU kernel makes verifying 16 tokens cost about the same as verifying 1. I ran out of time to build the actual engine, so I'm posting the idea, the numbers and a complete POC recipe (section at the end) in case someone wants to continue. The idea and the algorithm are what matter here. Who this is for People with 16/24/32 GB cards running models whose Q8 doesn't fit, for example 27–31B on 16–24 GB or 31B+ on 32 GB, who today use layer offload (llama.cpp -ngl/-fit, --n-cpu-moe, ktransformers). If Q8 fits in your VRAM, just use llama.cpp. I checked: Qwen 3.8 27B on a 5090 is a loss for this approach. The idea (borrowed from another system I'm working on, not related to LLMs) The idea comes from another system I'm working on, where a cheap 2-bit pass produces a pool of candidates and only that pool is re-scored exactly. There it worked better than doing everything at higher precision. Here it's the same pattern: 2-bit base in VRAM: randomized Hadamard rotation, 4 Lloyd–Max levels, GPTQ/LDLQ. Calibrated on chat data rather than C4, which alone was a ×1.37 boost to draft acceptance. On Polish prompts acceptance went from 1.22 to 3.03 tokens. Residual R = W − Ŵ in RAM: uniform 6/7/8-bit grid with GPTQ in the same rotated basis as the base. A uniform grid beat Lloyd–Max here, because the residual isn't Gaussian. Verifier = Ŵx (GPU) + Rx (CPU), batched over the draft tree. The CPU streams the residual from RAM once per check, so a 16-token tree costs about the same as 1 token. Draft = the same 2-bit weights, but attention reads the verifier's exact KV for already-verified positions. Only new positions use 2-bit KV. The draft costs zero extra memory because it's literally part of the verifier. I measure the draft by recall@P of the candidate pool (is the verifier's token among the draft's top P?) rather than top-1 accuracy. Results All results are greedy decoding with logits in fp32. KL is measured against bf16, and "top-1 mismatch" is the share of tokens whose top-1 differs from bf16. Everything was measured on the real models and real quantized weights (Qwen3.5-4B, Gemma 4 12B and 31B). For these runs the 2-bit codes and the residual were dequantized back into bf16 matrices, W_v = rot⁻¹(Q_base + Q_residual), and run in PyTorch/transformers. The math is identical to the GPU/CPU split (up to fp rounding), so precision and acceptance are real. What isn't measured yet is the speed and VRAM gain of the split itself. Verifier precision. Both setups are on a bf16 rounding floor. Model / setup bits/weight KL vs bf16 top-1 mismatch Qwen3.5-4B, int8 g32 (≈ Q8_0) 8.5 0.000327 0.99% Qwen3.5-4B, 2-bit + Q8 residual 2.125 VRAM + 8.125 RAM 0.000205 0.79% Qwen3.5-4B, bf16 ±1 ulp noise floor – 0.000237 0.85% Gemma 4 12B-it (chat), int8 g32 8.5 0.00167 1.12% Gemma 4 12B-it, 2-bit + Q8 residual 10.25 0.00087 0.58% Gemma 4 12B-it, bf16 ±1 ulp floor – 0.00089 0.55% On Qwen3.5-4B, the paired bootstrap 95% CI for KL against int8 g32 excludes 0. The Qwen3.5-4B numbers are on WikiText-2, 81,880 tokens. For Gemma 4 12B-it I used chat answers instead, because WikiText is chaotic for -it models. The residual knob (Gemma 12B, 32 chat texts): Residual (RAM part) bits/weight KL top-1 mismatch uniform 6-bit + GPTQ 6.125 0.0020 0.96% uniform 7-bit + GPTQ 7.125 0.00089 0.87% Q8_0-like int8 g32 (whole model) 8.5 0.0017 1.12% 6-bit gives Q8_0-class quality with about 28% fewer bytes read from RAM per check. 7-bit reaches the bf16 noise floor with about 16% fewer bytes than Q8_0. Draft quality vs Google's official Gemma 4 drafter. Both drafters are measured in tokens accepted per verification. Gemma 4 31B (doesn't fit 32 GB) 2-bit base 2-bit base + exact KV Google drafter (emulated) verifier's token in top-1/2/4/8 of the pool 0.769 / 0.886 / 0.938 / 0.969 0.853 / 0.951 / 0.983 / 0.996 0.812 / 0.907 / 0.950 / 0.970 chain, γ = 8 3.75 4.63 3.61 tree, 16 nodes 4.35 5.23 4.20 tree, 64 nodes 5.48 6.46 4.96 The 31B 2-bit codes take 7.78 GB versus 62 GB in bf16. On 12B the gap is smaller but in the same direction: chain 4.00 vs 3.34. The emulation of Google's drafter matches llama.cpp's measured acceptance exactly: 2.81 tokens per check at 3 draft tokens, 60% acceptance. So I trust the offline simulation for relative comparisons. Three-level draft (MTP head → 2-bit base → verifier): 5.14 tokens per check with ≤16 nodes and on average 1.43 base passes (31B). That's about the same as a 16-node tree from the base alone, with a draft roughly 5× cheaper. The algorithm is in the recipe below. CPU residual kernel (AVX-512, int8 tiles, 4.29 GB working set, Ryzen 9 9950X): tokens verified at once 1 4 8 16 32 ms per pass 108 127 116 125 204 It sits on the memory wall (33–38 GB/s measured) up to about 16 tokens, so a 16-node tree costs about the same as a single token. In llama.cpp offload a bigger draft got slower: MTP with 6 tokens ran at 9.78 tok/s versus 14.01 with 3. What was not done No end-to-end engine and no measured tok/s. The baseline to beat is Gemma 4 31B Q8_0 in llama.cpp with offload and MTP: 14.01 tok/s on my machine (5.84 tok/s without MTP). My estimate for this approach is ~30–40 tok/s at Q8_0-class precision. It's unverified. The main unknown is the GPU↔CPU synchronisation overhead across 60 layers. Prefill/TTFT is the big risk. A residual on the CPU is fine for decoding but bad for long prompts, so prefill needs its own design, for example streaming the residual to the GPU. Hardware caveat. My RTX 5090 runs at PCIe Gen3 x16 because of the host, and the RAM measures 33–38 GB/s. Both make every offload number pessimistic, including the baseline's. Scale of the tests. All acceptance numbers are greedy decoding on 32 chat prompts. What didn't work (to save you time) My original "surgeon" idea: keep a 2-bit base and add an exact correction R[:, S]·x_S for the top-k input channels by |x| per token. Results on Qwen3.5-4B, WikiText-2 perplexity (bf16 8.06, int8 8.06, 2-bit 89.5): 1% of channels gives 12.41, 2% gives 11.58, 10% gives 9.61. Helpful, but far from int8. The correction captures only 43–70% of the error energy at 1–10% of channels, and you need about 97%. Selection must be dynamic per token: a static channel set was much worse (16.5 vs 11.6). Selection must happen before the rotation: selecting after it was much worse (44.5 vs 12.4 at 1%). 3-bit base: a much better draft, but an int8-class verifier then needs a 6-bit residual, so it saves no bytes. Staying at 2 bits plus re-ranking was the better trade. bf16 residual: pointless, because it costs as many bytes as W itself, and a Q8 residual is already on the noise floor. Lazy per-token escalation of residual bits: no-go, because the weakest-link effect kills it. Closest existing work, and how this differs The closest is SubSpec (NeurIPS 2025). It drafts from a quantized copy of the same model, shares the KV cache with the target, and uses big trees under PCIe offload (Qwen2.5-32B FP16, 6.5 tok/s on a 4090). How this differs: SubSpec streams weights to the GPU; here the verifier is split by bits. The 2-bit part stays on the GPU, the residual is computed by the CPU in RAM, and only activations cross PCIe. SubSpec's draft is a separate 4-bit copy; here it's 2-bit and is the GPU half of the verifier, so it needs no extra memory. Measured result: a 2-bit draft with the verifier's exact KV beats a dedicated official drafter by 25–30%. The 6/7-bit GPTQ residual knob, and a CPU kernel that makes 16-node tree verification about as cheap as 1 token. Other related work: BitNest: W4 base plus a 4-bit residual in a rotated basis, all in VRAM. QSpec and QuantSpec: quantized self-speculation. DecDEC: residual in RAM with top-k channels, but computed on the GPU and quantized to 4 bits. HCInfer: approximate compensation on the CPU. Dovetail: verification on the CPU. Trees: SpecExec and Sequoia. Nested precision: Matryoshka Quantization and Any-Precision LLM. Staged speculative decoding: Spector & Ré, 2023. INT4 self-speculation in a shipping product: ipex-llm. As far as I could find, no single element is new. What seems new, at maybe ~70% confidence, is the combination plus the 2-bit measurements. POC recipe (everything needed to rebuild it) Algorithms used: Rotation: randomized Hadamard transform (Sylvester construction, block-diagonal, random ±1 signs, two-sided). Base: scalar Lloyd–Max quantizer (4 levels) with GPTQ/LDLQ error feedback from calibration Hessians. Residual: uniform symmetric quantization with GPTQ. Decoding: tree speculative decoding with a best-first tree built from rank statistics, plus a three-level (staged) draft. Notation: y = W·x, W is d_out × d_in. Quantize every Linear in the decoder layers: q, k, v, o, gate, up and down. Keep embeddings, lm_head, norms and vision/audio towers in bf16. 1. Rotation: randomized Hadamard transform (RHT), two-sided Hadamard matrix (Sylvester construction): start with H = [1] and repeat H ← [[H, H], [H, −H]] until the size is b, then divide by √b. The result is orthonormal and symmetric, so it is its own inverse. Block size: for a dimension n, b = the largest power of 2 that divides n, capped at 4096. H_b is applied block-diagonally: split the dimension into n/b blocks of b and multiply each one. Examples: 4096 → 4096, 8192 → 2 blocks of 4096, 15360 → 1024, 3840 → 256, 5376 → 256. Random signs: S_o (length d_out) and S_i (length d_in) are ±1 vectors, generated as torch.Generator().manual_seed(seed) + randint(0, 2) * 2 - 1. S_o uses seed and S_i uses seed + 1. I used seed = 20261008 + crc32(layer_name) % 100000; any fixed per-layer seed works, as long as you can regenerate it. The math: W~ = H_o · S_o · W · S_i · H_i # rotated weights (input side and output side) W = S_o · H_o · W~ · H_i · S_i # exact inverse x' = H_i · S_i · x # rotate the activation once per Linear y = W·x = S_o · H_o · (W~ · x') # rotate the output back The same in PyTorch (W is d_out × d_in, H_b is the b × b normalised Hadamard): def had_last(X, b): # block-diagonal Hadamard along the last dim return (X.reshape(-1, b) @ H_b).reshape(X.shape) def rot_fwd(W): W = had_last(W * s_i, b_in) # input side: W · S_i · H_i return had_last((W * s_o[:, None]).T, b_out).T # output side: H_o · S_o · (...) def rot_inv(Wr): W = had_last(Wr.T, b_out).T * s_o[:, None] return had_last(W, b_in) * s_i Rotation makes the weights roughly Gaussian and spreads outliers, so a 4-level Lloyd–Max grid fits well. Caveat: for dimensions like 5376 the mixing happens only inside 256-wide blocks. QuIP# instead uses a Kronecker product with a non-power-of-2 Hadamard, which might be better; I didn't test it. 2. 2-bit base: GPTQ/LDLQ in the rotated basis Calibration data: 28 × 2048 tokens of chat-templated conversations: 14 from UltraChat-200k, 8 Polish and 6 other languages from OASST2. Use in-distribution chat data, not C4. Hessian per Linear: H = Σ XᵀX / N over the Linear's inputs, in fp32. Rotate it the same way as x: H~ = H_i · S_i · H · S_i · H_i. Layer by layer: feed the next layer the outputs of the already-quantized layer. GPTQ settings: dead columns (diag ≤ 0): set H_ii = 1 and zero the weights; damping: add 0.01 · mean(diag) to the diagonal; upper Cholesky of H~⁻¹, block size 128, natural column order (no act-order). Scale: one per (output row, 128-column group) = RMS of the error-updated block, rounded to fp16 before quantizing. Codes: nearest of 4 Lloyd–Max levels, times the scale. To fit the levels, take ~2M rotated weights normalised by their group RMS, start from the quantiles 0.125/0.375/0.625/0.875, run 60 Lloyd iterations, then symmetrise. Fitted levels: ±0.4535, ±1.5066 (12B) and ±0.4511, ±1.5021 (31B). That's essentially the Gaussian 4-level Lloyd–Max. Storage: 2 bit-planes plus an fp16 scale per 128 weights = 2.125 bits/weight: 2.90 GB for 12B, 7.78 GB for 31B. 3. Residual (the RAM half of the verifier) R~ = W~ − Q_base, in the rotated basis. Quantizer: uniform symmetric b-bit, levels −qm..qm with qm = 2b−1 − 1. Scale = max|block| / qm per (row, 128-group), fp16. GPTQ on the residual: Hessian from the original model's inputs, damping 0.01, block 128. Bits: b = 6 gives Q8_0-class quality; b = 7 reaches the bf16 noise floor. 4. Verifier forward for one Linear (the GPU/CPU split) x' = H_i · S_i · x # GPU, O(d log d), once per Linear copy x' to CPU # n_tree × d_in floats y_b = Q_base · x' # GPU: 2-bit GEMV y_r = Q_resid · x' # CPU: b-bit residual from RAM, all tree tokens in ONE pass copy y_r to GPU # n_tree × d_out floats y = S_o · H_o · (y_b + y_r) # GPU The draft forward is the same but without y_r. 5. Draft attention with the verifier's exact KV For positions already verified, the draft uses the K/V that the verifier produced. Only new draft positions use 2-bit K/V. In an engine that's a single KV cache that the verifier overwrites after each check. To simulate it in transformers: replace the attention function (ALL_ATTENTION_FUNCTIONS["sdpa"]) with one that concatenates [K_exact ; K_draft] along the sequence and masks it. For query p, key j comes from the exact half if j < p − d + 1 and from the draft half otherwise, where d is the draft depth. Keep sliding-window and padding masks. 6. Building the draft tree Collect statistics. On held-out chat texts generated by the verifier, record at each draft depth d the rank of the verifier's token in the draft's logits. Rank is the number of tokens with a higher logit; 0 means argmax. Estimate probabilities. p_d(r) = frequency of rank r at depth d, for r < 16. Expand best-first. The priority of a path (r_1 … r_k) is the product of p_d(r_d). Pop the best path, push its children, stop at N nodes. N = 16 suits a memory-bound CPU verifier. Fit on half the texts and evaluate on the other half. Greedy acceptance. Follow the verifier's argmax down the tree. You accept the longest matching prefix plus 1 bonus token from the verifier. 7. Three-level draft (MTP → 2-bit base → verifier) This is a sketch of the logic I simulated offline. The settings: P = 2 base passes, k = 2 alternatives, A = 4, C = 2, D = 8, at most 16 nodes. m = MTP chain (up to 8 tokens) # cheap, model's own MTP head b = base top-2 at each position of m, ONE batched pass (exact KV for the context) passes = 1 for j in 0 .. D-1: if b[j].top1 == m[j]: main.append(m[j]) # both agree else: side.append(m[j : j+C]) # keep MTP's guess as a side branch main.append(b[j].top1) # trust the base if passes < P: # re-draft MTP from the base's token m = m[:j] + [b[j].top1] + MTP chain(...); b = base pass(...); passes += 1 else: break if j < A: leaves.append(b[j].top2) # base's 2nd choice as a leaf tree = main + side + leaves ( verifier checks it in one pass For Gemma 4 31B this gave 5.14 tokens per check with 1.43 base passes. Without the node limit it gave 5.36. 8. CPU residual kernel Layout: int8 weights as [d_out/16][d_in][16] (16 output rows interleaved), and fp16 scales as [tile][d_in/128][16]. x is [n, d_in] fp32. Loop: OpenMP static over tiles, keeping N zmm accumulators (one per token). Per input i, load 16 int8 values, convert to fp32, multiply by the scales, then do one FMA with a broadcast of x[t, i] for each token t. Settings: 16 threads, OMP_PLACES=cores, close binding. SMT and spread binding were slower. Packed 6/7-bit variant: unpack with vpermb + vpmultishiftqb (AVX-512 VBMI). I only verified correctness, not speed. 9. How to measure without fooling yourself Logits: compute them in fp32 from hidden @ lm_head. bf16 logits inflated every KL by about 0.0004–0.0005. Gemma 4 also needs softcap 30. Evaluation text: use chat texts. WikiText is chaotic for -it models, and greedy text from base models loops. Baselines: report KL and top-1 mismatch against bf16, next to the bf16 ±1-ulp noise floor and int8 g32. Offline emulation from rank statistics is reliable. It matched llama.cpp's measured MTP acceptance exactly (2.81). Gotchas Qwen3.5 MTP head: transformers drops the mtp.* weights, so rebuild the head yourself. Input = fc(cat[norme(emb(x{i+1})), norm_h(h_i)]), where h_i is taken after the final norm. With that wiring the depth-1 accuracy is 0.549; the reversed concat order gives about 0.009. For chaining, feed the MTP output after mtp.norm: depth-2 accuracy is 0.445, versus 0.343 before the norm. Gemma 4 full-attention layers: K = V (no v_proj), head_dim 512, sliding window 1024. Memory leak in the calibration loop: an exception used to stop the forward pass kept frames alive, about 3.5 GB of activations. gc.collect() fixes it. If you want to pick this up Build the engine: a 2-bit GEMV on the GPU (Triton) next to the packed CPU residual kernel; per-layer orchestration that overlaps the CPU residual with the GPU GEMV, keeping as much of the residual in VRAM as fits; verification of trees up to 16 nodes. Design prefill separately. Hybrid attention models (Qwen 3.8 27B: 48 of 64 layers are DeltaNet) need a recurrent state per tree node instead of an exact KV cache. MoE ideas, untested: load the residual only for experts with a high gate weight; the 2-bit base runs the same router, so it knows ahead of time which experts the verifier will need and can prefetch them. Test code tasks (HumanEval/MBPP) to check that the precision holds on syntax-heavy output. Setup: RTX 5090 32 GB, Ryzen 9 9950X, 192 GB DDR5, transformers plus custom attention for the exact-KV draft, and llama.cpp upstream for the baseline.   submitted by   /u/Happy_Guitar3521 [link]   [comments]