I bet you guys are familiar with embedding models such as Qwen3-Embedding or Gemini Embedding. They are dense embedding models used to search text across a large corpus. I am developing a RAG agent, and my biggest concern was local computing power. RAG largely depends on retrieval quality, which depends on model power. Plus, many use cases can't use an external internet connection or API models, so the embedding models have to run locally. But even 0.6B models are pretty heavy for small PCs. I don't mean your brand-new Mac Studio or DGX Spark. I'm talking about an 8GB RAM Windows PC at a boring desk job or your 5-year-old M1 MacBook Air. So I found this model called SPLADE. It is essentially sparse retrieval, like BM25, but uses semantic meaning. I wanna skip the part on how SPLADE works, because an LLM can explain it better than me. Anyway, the SPLADE model was so awesome and successfully replaced the dense embedding model (Qwen3-Embedding-0.6B). In my own experiment, I made an evaluation dataset with a few hundred queries. The corpus was all the documents on my Mac, which are mostly in Korean with some in English. As you can see in the graph, SPLADE outperforms the Qwen3-0.6B dense embedding model with similar query speed + much less memory. Due to its small size, it takes less than 1.5GB VRAM during inference. So it can run on a relatively small system like a MacBook Air. I tested both the dense embedding model and the SPLADE model on my M5 MacBook Air 16GB. The dense embedding model was almost impossible to use, and my MacBook soon rebooted due to high temperature. The SPLADE model made it till the end without any OOM errors or warnings. I didn't use the original SPLADE model; instead, I used KURE-sparse-nano, which has great performance in both English and Korean. The fun part is that hybrid retrieval with SPLADE & dense embedding is actually worse than SPLADE alone. If you are dealing with a low-resource local setup but want to run a fast & affordable local RAG system, I recommend trying a SPLADE model. It is powerful, but small. Plus, you don't need to re-implement everything because I open-sourced the whole implementation, so you can download and use it easily. Check out AutoRAG Agent. I optimized the inference of the KURE-sparse-nano model, its code, and the LanceDB indexing + inverted index implementation in Rust for faster search.   submitted by   /u/jeffrey-0711 [link]   [comments]