Follow-up to my earlier post. Where I ended up: a big DDR3 workstation board isn't worth the money when I already have two small 4-GPU boxes. So I tried making the two boxes act as one. TL;DR: I added a two-box layer split to Project Maya. Box A runs layers 0–23 Box B runs layers 24–44 plus the head. Only the hidden state crosses the network, ~25 KB per token. GLM-5.3-Flash (Maya-S24) runs at 38–42 tok/s decode, ~1,100 tok/s prompt reading at 30K, and 6–7 s per agent turn on 8× RTX 3060 spread over two old X79 boxes. It scores the same 48/48 on my coding-agent benchmark as a single 8-GPU box. Code: https://github.com/0xluk/project-maya/pull/1 (branch two-box-split on top of Maya v1.1.50; usage in docs/TWO-BOX-SPLIT.md) Setup 2 boxes, each an X79 board (Ivy/Sandy Bridge, DDR3, PCIe 3.0) with 4× RTX 3060 12 GB. No NVLink, no P2P, ex-mining rigs. RAM: 20 GB in Box A, 32 GB in Box B (more on that below). 1 GbE between the boxes, a plain switch. Project Maya v1.1.50 + my patch, Maya-S24 quant, 128K context, MTP speculative decoding on. GPUs capped at 130 W. One box is also clock-locked to 1500 MHz, because its PSU didn't like the unlocked load. How it works (short version) Box A (front) runs the OpenAI/Anthropic-compatible API, the tokenizer, the embedding and layers 0–23 on its 4 GPUs. Box B (back) runs layers 24–44, the MTP draft block, the output head and the sampler on its 4 GPUs. Per generated token, the residual (4×4096 values) crosses the link once. fp32 → bf16 → lossless pack is ≈25 KB, ≈0.3 ms on 1 GbE. Box B samples and sends back the token id, and MTP speculative decoding is pipelined across the link. Prompts are cut into 512-token pieces that flow through all 8 GPUs at once. Each piece crosses the link once (~12 MB), on its own thread. Security: the link has an IP allowlist plus an HMAC-SHA256 challenge-response in both directions, and the API needs a key. Size of the patch: ~1,900 lines, ~1,100 of them the link itself. Without --role the engine is unchanged. Results (1 conversation) Metric Two boxes (this) One box, 8× 3060 (rented, same quant) Decode, short 38–42 tok/s 42–45 tok/s Decode at ~30K context 33–37 tok/s not measured Prompt, 30K tokens ~1,100 tok/s ~1,050 tok/s Agent turn (~2.4K new tokens on a long chat) 6–7 s 5.0–5.3 s Coding-agent bench (5 real repo tasks, 48 hidden tests): Setup Score Time Two boxes, Qwen Code 48/48 22.9 min Two boxes, DeepSeek Harness 48/48 17.8 min 8× 3060 one box, Maya-S, Qwen Code 48/48 21.3 min Qwen3.8-Flash-Next on 2× 3060 (Strata), Qwen Code 48/48 23.2 min There were no malformed tool calls with either agent. One-shot hard suite (10 tasks, one request each, no tools, 32K output cap including thinking, reasoning high): 6/10 full passes, 67% of tests. For comparison: Opus 5.5 9/10, the same quant on the 8-GPU box 7/10, and Flash-Next 6/10. Passed: json_patch, fixbug_calc, event_scheduler, csv_rfc4180, c_utf8, singleflight_cache. 3 of the 4 misses are budget, not wrong code: on regex_engine, lisp_interp and c_bigint the model was still thinking when it hit 32K tokens, so no code came out at all. The real miss is c_json_canon: 22/31 tests passed, with the misses on Unicode escape handling. Next: rerun the three capped tasks with a 64K cap, which the 128K context allows. What was hard RAM, not the network, was the real bottleneck. At 16 GB per box, the experts that didn't fit in VRAM came off a cheap SATA SSD on every prompt chunk: 31 s turns, 173 tok/s on a 30K prompt. Going to 20 + 32 GB made it 0 SSD reads, 6–7 s turns and 1,100 tok/s. The network costs ~1–2% of decode and ~3% of a turn. A correctness scare that wasn't a bug. Compared with a single-process reference, the split showed KL ~1.6e-2 and ~95% top-1, and I spent a while hunting a split bug. It turned out a single process on one machine moves by the same amount (KL 1.1–2.5e-2) just from changing the RAM tier size or the prompt chunk size. Near-ties in the top-8 expert choice flip on rounding noise. The split is inside that spread. Hardware gremlins: A new PSU powered off under 4×130 W unlocked, so that box stays clock-locked. Two GPUs once came up at PCIe x4 instead of x8/x16 after a power-loss reboot. I added a boot check that refuses to start the engine on a degraded link. Tuning that mattered: 512-token prompt pieces so all 8 GPUs overlap: ~1 s off each turn. Fewer layers on the last GPU, which also carries the head and the MTP block. Keeping the whole KV cache in VRAM. What didn't help: bigger prompt chunks, which push the 20 GB box into SSD reads. What I skipped: a 10 GbE NIC. It would sit in a PCIe 2.0 x1 slot, and the link isn't the bottleneck anyway. What's next A 5th 3060 in one box, on an x1 riser. Then all experts fit in VRAM, which should get turns to ~5 s. Rebasing onto the current Maya release. Happy to answer questions. If someone has two boxes with 24 GB cards, I'd love to see numbers.   submitted by   /u/alexlucaci [link]   [comments]