I was getting ready to sell an old kiosk Mini at our office that was getting an upgrade, and figured I would see what happened if I tried to make it do something it by all rights shouldn't be able to. I loaded up Strata and ran the iQ3_xxs quant of QFN with a max of 128k context. I used opencode on my normal workstation to build a script that would benchmark the performance and output quality of typical tasks I do (mostly large csv sorting and low end code grunt work)at varying levels of context. I have to say, it worked far better than I would have expected it to. I assume that the speed is always at its worst (as if it had full context) because it never has a chance to cache anything and is always re-reading and swapping experts across prompts, but as an aside it means that token speed for both prefill and decode are the same regardless how much you're making it do. The median speed between all tests from 16k to 120k context was 77 tok/s prefill and 5.5 tok/s decode. That means it could actually stand in as a barely passable chat generator, and is not hopeless as a "living on dial-up" code helper of last resort. It does appear that it is pretty rough on the SSD, but the write load is not much worse than someone using a Mini as a pretty active DB server or other "proper" server task in an operating office environment.   submitted by   /u/SocietyTomorrow [link]   [comments]