been working on a local desktop reading app for the past few months and hit some insane hardware walls trying to get everything to run on my laptop. thought id share the architecture in case anyone else is trying to run heavy models on consumer hardware. my setup is an RTX 3050 with only 4GB of VRAM and 16GB of system ram. the stack is a tauri (rust) shell, a nodejs orchestrator, and a python sidecar for the pytorch models (bge-m3, bge reranker, and kokoro tts). here's how i stopped it from melting my laptop: the node memory cliff early on, the node heap spiked to 14.6GB during ingestion because it was firing off too many async promises at once. basically oom'd the whole system. i fixed it by implementing a strict sequential await barrier (f65). it forces the ingestion stages to stream chunk by chunk instead of all at once. peak node ram dropped from 14.6GB down to ~296MB. processing 500k words takes about 5.9 mins. 4gb vram limit loading bge-m3 in full precision instantly ate like 3.9GB of vram, leaving nothing for the reranker or tts. switched to fp16 which dropped it to ~1.1GB. but the real fix was an idle lifecycle manager (f43). if a model sits idle, it evicts it from vram to free up space. it reclaims about 3.2GB of idle vram so the system doesn't choke. windows path traps if you're using rust and node together on windows, beware of verbatim paths (\\?\C:\...). passing those into node 22 fs.lstat throws fatal EISDIR errors. had to write path sanitization wrappers on the rust side. kept node as the sole sqlite WAL writer too so python and rust dont cause db lock contention. it's open source if anyone wants to look at the code or roast my architecture lol. repo is here: https://github.com/ayumu-ml/Omnitome happy to answer any questions about the vram management or the rust/node/python boundary   submitted by   /u/ALexiosK11 [link]   [comments]