Hello hello 👋 , first post here, and it's self-promotion, so feel free to ping me to remove it if it violates rule 4. Sorry in advance. A friend and I started Gufo as a pet project (MIT license) to get the most out of the AMD Strix Halo chip, while supporting a few best-in-class models. We released it almost a month ago, and it has been an amazing journey. I won't bother you with benchmarks, contribution numbers, etc. You can take a look at the GitHub project if you want, especially since many people in r/LocalLLaMA don't even own a Strix Halo. Gufo is teaching us a lot, and besides the project itself, we'd like to give back to the open-source community with a series of blog posts explaining some of the concepts and design decisions behind an AI inference engine. This is the real value for this community: you don't need a Strix Halo to find these articles useful and interesting, because they cover topics common to many other projects. Our first blog post is about caching, and how we built our own solution to take advantage of this chip's peculiar characteristics: a lot of unified memory, but lower bandwidth. How Gufo caches a hybrid model   submitted by   /u/fedeizzo [link]   [comments]