If you're running 27B at BF16, you will save VRAM and gain speed by doing this (bonus uncensored variant info included as well): Opus 5.2 explains: Summary A lossless packed weight format (BF16_P12). The same weights, bit for bit, stored in 12.25 bits each instead of 16. The model takes 11.4 GiB less graphics memory and generates about 20-24% faster, because generation speed is limited by how fast weights can be read from memory. A working setup around it: both cards in tensor parallel, the model's built-in MTP layer for speculative decoding, several long chats at once, vision, and encrypted chat checkpoints. The "uncensored" variant as a 10 MB add-on instead of a second 52 GB model, switched on per request. BF16 (before) Packed (after) Weights on the card 50.9 GiB One card, plain 29.5 tok/s Both cards, plain 48.8 tok/s One card + MTP (3 ahead) ~54 tok/s Both cards + MTP (3 ahead) ~77 tok/s Weights changed — The machine and the model Two NVIDIA RTX PRO 6000 Blackwell cards (96 GB each, limited to 450 watts), Linux, llama.cpp (current upstream plus our patches). Qwen3.8-27B: 64 blocks (48 linear-attention, 16 full-attention) plus an MTP block used only for drafting. The BF16 GGUF is about 52 GB in 866 tensors. Chat memory (KV cache) costs 64 KiB per token, so one 256K-token chat is 16 GiB. That is why memory freed from the weights matters: it turns directly into more or longer chats. We did not want Q8 or smaller: in our use the quality loss was noticeable. The goal was to keep the exact weights. The idea behind the packed format A BF16 weight is 1 sign bit, 8 exponent bits and 7 mantissa bits. The sign and mantissa look random and cannot be compressed. The exponent can: neighbouring weights have nearly the same size, so within a short run of weights the exponents fall in a narrow range. Early tests on the checkpoint showed this clearly. The low byte of each weight has 7.97 bits of entropy (incompressible), the high byte only 2.68. The format: BF16_P12 Weights are packed in blocks of 256 weights from one row, 392 bytes per block (12.25 bits per weight). Every block is self-contained, so any block can be decoded without reading anything else. Per weight: one byte holding the sign and the 7 mantissa bits, plus a 4-bit code for the exponent. Per block: a base exponent. Code c means "base + c", so a block covers 16 consecutive exponents. A weight whose exponent falls outside that range is listed in the block header (its place and its true exponent). A block can list up to 3 weights. A tensor with a block needing more stays plain BF16. There is exactly one packed form for any block, so packing is deterministic. What the real file looks like: About 1 weight in 17,000 needs to be listed. 98.5% of blocks list none. Only one tensor does not fit: the token embedding table (2.37 GiB), which stays BF16. Result: 50.88 GiB of BF16 matrices become 39.51 GiB, 11.37 GiB less. The small 1-D tensors are F32 and are left alone. We first considered a 13-bit layout with three separate streams. The scan of the whole file showed the 12-bit block layout saves more and is simpler to decode. How the engine uses it The packed data is never unpacked into a full copy of the model. Which path runs depends on how many tokens are processed in one step: 1-4 tokens per step (normal generation, and generation with MTP drafts): the multiply reads the packed blocks directly and rebuilds each 16-bit weight in registers just before using it. 5-16 tokens: a copy of the engine's BF16 tile kernel that decodes while loading. Results are bit-identical to the BF16 kernel. More (reading a prompt): a cache-sized piece of the matrix is unpacked at a time and handed to cuBLAS, as BF16 would be. There is also a "proof mode" (an environment switch) that always unpacks first. It is slow but gives the unpacked model's results bit for bit, which makes it a useful reference. Size of the change: a new tensor type, a processor fallback, the CUDA kernels and file-format plumbing, about 940 lines across 17 files, kept as its own patch on top of upstream. A separate converter turns a BF16 GGUF into a packed one. How we checked it Round trip: the converter unpacks every tensor it has packed and compares it with the original before writing. It can also verify an existing packed file against its original. Both the normal and the uncensored model pass with every tensor exact. Synthetic values: the reference packer was tested on made-up data covering unusual exponents, not only on the model. Kernel against exact arithmetic: for real matrix shapes the packed kernel's result was compared with a double-precision reference. It agrees to about 1 part in a million, the same as the BF16 kernel; any difference is the order of additions, not the weights. Same answers: five questions with randomness off. Proof mode gave the same text as BF16 for all five. Normal mode also gave the same text for all five. Everything else still works: MTP drafting, vision, both-cards mode, the add-on, and saving and restoring chat checkpoints (the reply after a restore is identical). Speed and memory in detail Three prompts, 512 tokens each, thinking-mode sampling. Setup BF16 tok/s Packed tok/s BF16 memory Packed memory One card 29.5 36.7 54.0 GB 42.5 GB One card + MTP 57.5 / 53.4 / 51.2 57.6 / 45.7 / 50.7* 55.7 GB 44.0 GB Both cards (tensor parallel) 48.8 58.6 27.8 GB each 22.1 GB each Both cards + MTP 76.8 / 76.1 / 73.3 91.4 / 73.0 / 80.9* 28.7 GB each 22.9 GB each * With MTP the speed depends on how many drafted tokens are accepted, which varies with the text, and these two runs did not generate the same text. On identical text (randomness off) the packed model was 7-8% faster with MTP on one card. Things worth knowing: Why MTP gains less than plain generation: with drafting, each step checks several tokens at once, so more time goes into work that is not limited by reading weights. Reading a prompt in the engine's default pieces of 512 tokens was about 20% slower than BF16 (2,570 against 3,150 tokens per second), because weights are unpacked piece by piece. With pieces of 2,048 tokens it matches BF16's speed at 512, for 0.6 GB more memory. We run it that way. Loading is slightly quicker (4 s against 6 s) because the file is smaller. What the freed memory buys: on both cards, up to 6 chats of 256K tokens each (4 before). On one card, the weights plus two 256K chats fit with room left over. The rest of the setup Tensor parallel over both cards gives about 1.6-1.7x one card without drafting. The cards have no direct link to each other, and it still pays off. MTP drafting: Qwen3.8 ships with a small extra layer that guesses the next few tokens. The main model checks the guesses, so the output is the same as without it. Three tokens ahead gave the best speed, about 1.9x on BF16. Several chats at once share one pool of chat memory, each chat limited to 256K tokens. Sampling follows the model card for each mode (thinking on: temperature 1.0, top-p 0.95, top-k 20; thinking off: 0.7 / 0.8 / 20 with presence penalty 1.5). The thinking level (low / medium / xhigh) is passed through to the chat template. The uncensored variant as an add-on There is a community variant of the model with refusals removed (thanks to OrcaRouter.ai). Comparing its weights with the original showed the difference is one direction removed from certain matrices, which can be written exactly as a low-rank add-on (LoRA). So instead of a second 52 GB file we keep a 10 MB add-on, loaded but not applied. A request that names the uncensored model turns it on for that request only. It costs about 3% speed while in use (35.5 against 36.7 tok/s packed) and nothing otherwise. It works with the packed weights: the add-on is applied alongside them, the packed data is not touched. Limits and open points Not tried yet: a separate draft model (DSpark / DFlash) in place of MTP. One exists for this model; whether it beats MTP here is unmeasured. The format was designed from this model's statistics. Other BF16 models should pack similarly, but we have only measured this one and its uncensored variant.   submitted by   /u/EitherMarch1255 [link]   [comments]