Been dealing with the same thing a lot of people here probably are - RAM prices keep climbing and buying a single machine with enough memory for the models I actually want to run just isn't realistic right now. So instead I hve been working on pooling the RAM I already have sitting around across a few devices instead of buying more. Latest test: loaded a 30B model split across a mini PC (12GB) and a Mac mini (5.5GB), getting mid 30s tokens/sec at around 1 second latency - better than I expected for a model this size split across two machines over plain lan. The part I actually wanted to test was the knowledge base side, not just the model. Asked it about a recently approved drug for a rare condition - it had no idea, made something up. Fed it a real research PDF, watched it split the document and store it in RAM across both devices (not written to disk), then asked the same question again. It got it right that time, pulling straight from the fed document. One thing I found genuinely useful: the device running the model and the device storing the knowledge base don't have to be the same machine. So your strongest box can focus on inference while a weaker device just holds document shards in the background. There is also a recovery step if the device holding a shard drops offline - it pulls the backup copy back and re-splits it. Repo: https://github.com/trademav/ramdeck-core-public Curious if anyone else here has tried keeping a RAG index fully in RAM across multiple machines instead of a vector DB on disk - wondering how that tradeoff holds up with much bigger document sets than what I tested.   submitted by   /u/Medicine_Blogscanner [link]   [comments]