Follow up to my post about the £300 Snapdragon mini PC: https://www.reddit.com/r/LocalLLaMA/s/IXZDyyYFDl Quick recap of how I got here: I started on llama.cpp, then pretty quickly moved over to Janas (github.com/prabanta-dev/janas). I ported it to ARM and contributed the work back as PRs (NEON kernels, a Windows on ARM build, and a cache bug fix Maurizio has since shipped in his own release). Working inside Janas gave me a load of ideas, so I'm now writing my own engine, codename **ToRc**. It's not released, YET, but it's what I used to produce the numbers below. Hardware: - Lenovo IdeaCentre Mini: Snapdragon X X1-26-100 (8 Oryon cores, Hexagon NPU; the Adreno GPU isn't used), 32 GB LPDDR5, the stock 512 GB NVMe (the models stream from it), Windows 11. I paid just £299.99 each, prices vary but under 400 is possible on eBay etc fairly easily and they are still under 400 at Amazon. - "Two boxes" = two of them (£600) joined by one USB4 cable. - No GPU anywhere. ToRc uses the CPU(s) cores, plus the NPU(s) for some of the dense maths. Same two boxes, all figures from today, nothing else running. The models: - **Qwen3.6 35B A3B**: bartowski `Qwen_Qwen3.6-35B-A3B-Q4_0.gguf` (19.4 GB), with its MTP head (Q4_0). - **Qwen3.8 Flash Next IQ4_NL**: ggml-org `Qwen3.8-Flash-Next-IQ4_NL` (95 GB). - **Qwen3.8 Flash Next UD-Q4_K_XL**: unsloth `Qwen3.8-Flash-Next-UD-Q4_K_XL` (104 GB). - Flash Next MTP head: bartowski `mtp-Qwen3.8-Flash-Next-Q8_0.gguf` (3.9 GB). - llama.cpp (b11700, CPU, ARM64) runs the GGUFs; Janas and ToRc run the same files converted to Janas's .jns format... I have a new file format I'm working on but that's what I used to test today 👍 The two payloads (identical for every engine): Title numbers: two boxes, read/generate on the bench payload below (a ~580-token English prose prompt, 128 tokens generated). Every engine gets byte-identical requests through its own OpenAI-style chat server (llama-server, janas-server, torc-server), timed from the client. Greedy (temperature 0), thinking off. **Bench (prose, ~580 tokens in, 128 out)**: one user message: the opening of *Alice's Adventures in Wonderland* chapter I (Project Gutenberg #11, first 6 paragraphs, 2400 characters, ~580 tokens with the chat template) + "Continue the story.", 128 tokens out. **Read** = prompt tokens / time to first token (so it includes the request overhead and first token). **Write** = tokens / generation time. Each run starts with a different first line so no server gets its prompt cached from the run before. A warm-up run, then the average of two runs. Read is each engine's better configuration of the two (plain or MTP; ToRc two boxes uses 128-token prompt blocks). **Real chat (code, 62 tokens in, 550-970 out)**: "Write a Python function that parses an ISO 8601 duration string like P3DT4H12M5S into total seconds, with input validation and docstring, plus 5 pytest test cases. Code only, no explanation." (62 tokens with the template), up to 1500 tokens out. Reply tok/s, then I ran the 5 tests each model wrote, plus the same 6 reference checks on every model's function (P3DT4H12M5S, PT1H30M, P1W, P2D, and two bad inputs that must raise). Qwen3.6 35B A3B Read / Write / Write + MTP: the prose bench (~580 tokens in, 128 out). Coding chat: the code payload (62 in, 550-970 out). Qwen3.6 35B A3B (read / write / write with MTP, then coding chat plain / MTP, then tests): llama.cpp, 1 box: 133.8 / 25.8 / 20.6. Coding 27.7 / 39.3. Tests 4/5, checks 6/6 llama.cpp RPC, 2 boxes: 107.1 / 24.9 / n/a. Coding 26.9. Tests 4/5, checks 6/6 Janas, 1 box: 169.5 / n/a* / 36.2. Coding 42.9 (MTP). Tests 5/5, checks 6/6 ToRc, 1 box: 228.0 / 43.4 / 49.5. Coding 42.6 / 53.3. Tests 5/5, checks 5/6 ToRc, 2 boxes: 315 / 43.3 / 43.3. Coding 42.6 / 59.6. Tests 5/5, checks 6/6 - *Janas (my update 4 build) uses the 35B file's built-in MTP layer even with `--no-mtp`, so it has no plain number: its "MTP" figures are the average of all its settled runs (bench 33.7-37.8, coding 41.3-45.4). - Engine-internal (ToRc's own bench, no server in the way): two boxes read **343.6 tok/s on a 600-token prompt**, and 349.4 on 512 tokens. The 315 in the title and table is the same engine timed from the client like every other engine, so it includes the HTTP request and the first token. - On two boxes ToRc uses both for the prompt, then pulls the state back and writes the reply on box 1 alone: generating across the cable costs a round trip per token, more than the second box saves. So plain writing is the same on one box and two (42.6 on the coding chat, byte-identical replies). With MTP the two runs wrote different replies (626 vs 968 tokens), so 53.3 vs 59.6 comes from the text, not the box. - MTP depends on the text. On the Alice prose llama.cpp's MTP was slower than plain (25.8 -> 20.6, drafts mostly rejected), ToRc one box gained 14% (43.4 -> 49.5) and two boxes gained nothing. On code they all gain: llama 27.7 -> 39.3, ToRc 42.6 -> 53.3 (one box) and 42.6 -> 59.6 (two boxes, same reply text both ways). - The tests: every engine runs the same model, but tiny numeric differences change a greedy reply, so each engine's reply is different code. llama.cpp's one failed test is the model's own arithmetic (it says P3DT4H12M5S is 300725 seconds, it's 274325), its function is right. The ToRc 5/6 is a reply that doesn't accept weeks (P1W, valid ISO 8601). And now the headline! Flash Next! Unfortunately Flash Next wouldn't run on Janas, which is why I started on my (hopefully soon to be released) engine, ToRc. Flash Next IQ4_NL is 95 GB. One box has 32 GB of RAM, two boxes 64 GB, so it never fits: the experts stream off the NVMe as they're needed. That's where the engines split apart. Read / Write: the prose bench (~580 tokens in, 128 out; ToRc two boxes with pipelined MTP, everything else plain). Coding chat: the code payload (62 in, ~750 out). Flash Next (read / write, then coding chat, then tests): llama.cpp IQ4_NL, 1 box: 7.3 / 4.2. Coding 3.2. Tests 5/5, checks 5/6 llama.cpp RPC IQ4_NL, 2 boxes: 10.9 / 5.8. Coding 5.0. Tests 5/5, checks 5/6 ToRc IQ4_NL, 1 box: 10.7 / 6.6. Coding 4.6. Tests 5/5, checks 6/6 ToRc IQ4_NL, 2 boxes: 46.2 / 20.3. Coding 19.2. Tests none written, checks 6/6 ToRc UD-Q4_K_XL, 2 boxes: 34.3 / 18.3. Coding 16.4. Tests 5/5, checks 5/6 - llama.cpp needs `--no-repack` here: on ARM it repacks IQ4_NL weights into RAM and 95 GB won't fit, so without it the model doesn't load at all. Its RPC server keeps its share of the weights in RAM (it can't stream from disk), so box 2 takes 12 of the 48 layers and box 1 maps the rest from its NVMe. - ToRc on two boxes generates across both: box 1 drafts the next tokens with the MTP head while box 2 is still finishing the current one, so the cable wait is mostly hidden. One box runs without the MTP head: with the experts coming off the SSD, MTP adds nothing there. - The Q4_K_XL is the bigger, better quant (lower perplexity than the IQ4_NL on a 3800-token Delphi code test) and costs ~10% speed. - The IQ4_NL two-box coding reply wrote a correct function but no tests, so "none written". Its function passes all 6 reference checks. Power and cost: No GPU anywhere in this: two mini PCs, £600 all in. Getting ~100 GB of model into GPU VRAM costs many times that before you've paid for the electricity. These won't beat a GPU on speed, obviously, but cost and speed... This has been an interesting little project IMHO 🙏. A 100 GB-class model at chat speed, on two small, quiet boxes that consume around 35W each at load (proper measurements to come) Anyway, this was a fun week long project and the first time I've used a snapdragon let alone tried to run an LLM on one... ToRc isn't released yet, I want it tidier and properly tested first. What's next on it: - Pipelined speculation for the 35B on two boxes (today the second box only helps with reading; it already works for Flash Next). - Flash Next on one box is limited by the SSD, not the CPU: a second NVMe with a copy of the experts might help by splitting every read across two drives. - Testing more models through the same path, maybe, but for my use cases, these two are probably enough 👍 Happy to answer questions or run something you want to see if I get the chance... Thoughts and feedback welcome 🙃👍   submitted by   /u/WorldlyAd7946 [link]   [comments]