I got into local AI earlier this year and started messing around with different models and hardware. That's how I came across tiiny. I really liked the design and the whole idea behind it so I ended up becoming one of the early backers. After what felt like forever, mine finally showed up yesterday. So far, it's pretty much what I expected. I've spent afew hours trying out different models and running some performance and coding tests, and figured I'd share what Ive found. This is my first time putting together a benchmark post like this, so sure there are things I might miss or could do better. If you see anything off with my testing or think there's something I should further include, feel free to let me know. really appreciate any feedback from you guys. https://preview.redd.it/fcl01pkkmszh1.png?width=959&format=png&auto=webp&s=7bfc80b336b7baef2373c01fdb4ee12404e9555b Hardware & Test Setup what really sold me on Tiiny was the KS price, which seemed like a nice deal for the hardware, and how portable the whole thing is. It's just about 5.6 × 3.1 × 0.9 inches and weighs around 300g. For anyone unfamiliar with the specs, Tiiny Pocket has 32GB of SoC memory and 48GB of dedicated NPU memory. That's 80GB in total, though it's not unified memory. Tests were collected on TiinyOS 0.1.36, Here are the models I tested so far. Model Architecture Quantization Model Size Qwen3.5-9B-Turbo Dense, 9B INT4 6.55 Qwen3.6-35B-A3B-Turbo MoE, 35B / 3B active Mixed INT4/INT5 17.73 gpt-oss-120b MoE, 117B / 5.1B active Mixed INT4/INT8 67.30 Performance Results For the speed tests, I used the same benchmark text with ~4K input tokens, prompt caching disabled, and five runs per model. The numbers below are medians. Model Input / Output Tokens PP (tok/s) TG (tok/s) TTFT Total Time Qwen3.5-9B-Turbo 4,074 / 97 1,041.30 75 4.05s 5.45s Qwen3.6-35B-A3B-Turbo 4,074 / 139 590.6 63.6 7.04s 9.28s gpt-oss-120b 3,868 / 390 458.4 15.97 8.78s 33.07s 9B was the fastest, which wasn't too surprising. What caught my attention was the 35B MoE model still managing around 64 t/s. 120B was noticeably slower, averaging around 16 tok/s. It took over 30 seconds to complete the benchmark request, although it generated more tokens than the other models. A Coding Test I also wanted to see how it handles real-world coding tasks. I gave the models the same task: create a small Python inventory and order management package across four files, covering stock updates, validation, and order handling. I used the same prompt for all three models, with low thinking/reasoning settings and a maximum output of 12,288 tokens. How I checked the results: the generated code was tested separately on a Mac using Python 3.12, with 10 automated tests covering the main requirements, including inventory consistency, order idempotency, input validation, and CLI behavior. A full pass meant all 10 tests succeeded. I ran this more than once and kept the best run per model; the table shows that run. Model Input / Output Tokens Total Time Result Qwen3.5-9B-Turbo 703 / 5,977 104.32s Two attempts, neither gradable Qwen3.6-35B-A3B-Turbo 741 / 11,807 235.09s 9/10 gpt-oss-120b 763 / 3,774 228.54s 10/10 9B never produced a gradable answer: the first attempt ran into the 12288 output cap, and the second generated inventory/cli.py twice, so there was no single version to test. Taking one of the copies, it would have been 5/10. 35B got 9/10 with 1,768 code tokens. A bit under the 2-4K I was aiming for. The only failure was a pretty specific edge case, if an order had already sold out the remaining stock, retrying that same order would fail instead of returning the original receipt. Gpt 120b was the only model passed all 10 tests, with 2,594 code tokens, in about 229 seconds. The interesting part is that 35B and 120B took almost the same time: 229 seconds versus 235. So while 120B was the only one to get everything right, 35B wasn't far behind, and both managed to finish. Results & Takeaways 9B was the fastest overall. At 75 tok/s with a ttft of around 4 seconds, it's probably where I'd start for quick, lightweight tasks. It did struggle with the more complex test, though. 35B for me is the sweet spot for Tiiny.At around 64 tk/s, it's surprisingly fast and it handles a good number of the tasks I've tried pretty well. It still has its limits with more complicated coding tasks, as the multi-file test showed, but overall it keep a pretty good balance between speed and capability. It's actually the model Ive been using the most so far. 120B actually works, but you need patience. The multi-file coding result was impressive, especially compared with the smaller models in this one test. But 16 tk/s is slow enough that I wouldn't want to use it for every little question. It also took around 75 seconds to load in this run and used almost all of the dNPU memory. So far Id lean toward 9B or 35B for faster interactions and keep 120B for occasional, more demanding tasks where waiting a few minutes isn't a big deal. TL;DR Tiiny can run some fairly large local models in a small device, and the performance of smaller models (9B/35B) is nice. The 120B result shows there's potential for more complex work, but the speed and reliability trade-offs are real. However, my biggest gripe so far is the limited model selection. Right now, you can't import your own models or install something from Hugging Face. The tiiny team did mention on Discord that they're planning to release a model import tool, though. Once that's actually available, I'd love to try out some of the other models I've been wanting to run and share the results here. Overall Im happy with it, and for the price I got it at, that amount of RAM and SSD would be much more expensive today. I think I saw the projected retail price is almost double the KS price.   submitted by   /u/IronDiligent6698 [link]   [comments]