This is going to be a long one. TL;DR: I’m building a local-first AI environment rather than continuing to rent all of my inference from cloud providers. My desktop has 2× 5060 Tis running Occamy 1.0 at ~100 tok/s decode with two ~175K-context worker pools, while a 96 GB M5 Ultra will serve as my primary offline inference appliance running Qwen 3.8 Next Flash. OpenChamber is the persistent workspace, accessible from my desktop, laptop, and phone, with a ZDR API available only when I deliberately want a larger cloud model. I paid $800 total for the GPUs before prices increased, lease the Mac Studio for $117/month, and use a $5/month VPS for controlled off-network connectivity. The Mac will be VLAN’d, LAN-accessible, and WAN-denied. The bet isn’t that local hardware beats cloud pricing today. It’s that owning a permanent inference floor, combined with future improvements in open models, quantization, inference engines, caching, and agent workflows, will make the same hardware increasingly useful over the next three years. The be is that this will be insulating me somewhat from future API/subscription pricing, restrictions, and provider changes. I've spent the last several months moving toward local inference as my default and cloud inference as the exception. I'm getting close enough to the finished architecture that I figured I'd write up what I'm doing, why I'm doing it, and the assumptions I'm making. A few caveats first because I know where some of the comments will go: This is expensive by normal-person standards. You can absolutely get access to more powerful models for dramatically less upfront money through subscriptions and APIs. If your goal is simply "I want access to good AI," I am not arguing that everyone should go buy a pile of hardware. My goals were different. I wanted: Constant access to one master workspace from desktop, laptop, and phone Private inference by default Fast inference Multiple concurrent agents/workers Enough context to make agentic workflows actually useful The ability to keep my primary inference hardware isolated from the internet A fixed level of AI capability that I own/control instead of continuously renting A reasonable chance that the same hardware becomes more capable as models and inference software improve I'm essentially betting that buying/locking in compute today will age well over the next three years. I'm also betting against subscription/API pricing becoming universally cheaper forever, and against inference hardware becoming so dramatically cheaper in the next 2-3 years that buying today looks ridiculous. I could absolutely be wrong about either of those things. I'm not trying to predict that today's models will remain competitive with frontier cloud models. I'm locking in the ability to run roughly today's level of local intelligence indefinitely, while betting that future open models, quantization, inference engines, caching, sparse attention, MTP/speculative decoding, agent harnesses, etc. will let the same hardware do increasingly useful things. Hardware My worker box is my existing desktop with: 2x RTX 5060 Ti I bought them for about $400 each, before GPU pricing went nuts. So my actual cost basis is about $800 total. Last I looked, recreating that purchase today would be considerably more expensive. They're currently running Occamy 1.0 at around 100 tok/s. More importantly for what I'm doing, I have enough memory available to maintain approximately two independent ~175K context pools, so I can have two substantial subagents working concurrently rather than serializing everything through one context. The GPUs are mine permanently. Even if the rest of this experiment changes, I expect them to remain useful worker inference. Primary inference: M5 Ultra Mac Studio, 96 GB This arrives next week. I'm leasing it for $117/month through Apple's upgrade program rather than buying a huge-memory machine outright. Over three years that's $4,212 in lease payments before any potential buyout. That was intentional. I basically have three exits at the end of the three-year window: If AI hardware/model requirements haven't changed dramatically and 96 GB remains plenty useful, I can buy it out. If local AI advances massively and I suddenly want 256/512 GB or something completely different, I can upgrade instead of being stuck trying to justify an enormous sunk purchase. If the economics of local inference stop making sense entirely, I can exit. I'm comfortable locking myself in for three years. I'm much less comfortable pretending I know what local AI hardware requirements will look like six years from now. The Studio's planned primary model is Qwen 3.8 Next Flash, probably through oMLX. For now, while I'm waiting for it, DeepSeek v4.1 Flash via API is filling the primary-agent role. The Studio will not be a general-purpose computer. It is an appliance. It will sit in my rack, on its own VLAN, LAN-accessible and WAN-denied. No browsing Reddit. No downloading packages whenever it feels like it. No agents getting curious and wandering onto the internet. It receives inference requests and returns inference results. That's it. Physical setup I actually moved my main desktop downstairs next to my rack. My "desktop computer" upstairs is now basically a tiny N150 thin client running Fedora KDE. That's turned out to be a benefit rather than a compromise. My office is quiet, the N150 uses almost no electricity, and all the hot/noisy/high-power hardware gets to live downstairs. Given that local inference isn't exactly electricity-friendly, I'll take the savings where I can get them. Infrastructure I already had a homelab, so some of this infrastructure existed independently of AI. I have: A Proxmox cluster for VMs/LXCs on demand UPS protection across the important infrastructure NUT handling graceful staggered shutdowns Nonessential systems shutting down early during an outage Important infrastructure staying alive as long as practical Nightly backups of critical systems Automated restoration testing where practical That last one is important to me. A backup job completing successfully tells me that I successfully created a backup. It doesn't necessarily tell me that I can restore it. For the important stuff, I have an agent kick off restoration validation after the backup completes where it makes sense to do so. VPS / off-network connectivity I also have a small Contabo VPS that costs me $5/month. I believe that particular pricing tier may be legacy now, so don't take that as "you can go buy this exact VPS for $5 today." Its purpose isn't compute. It's essentially a controlled connectivity point between my home infrastructure and the outside world. That gives me remote access while letting me tightly control what can come into or leave my home network rather than exposing the inference machines directly. Software At the moment the important pieces are: OpenChamber OpenCode llama.cpp oMLX once the Mac Studio arrives I'm not particularly religious about inference backends. llama.cpp is useful today. oMLX appears to be the right answer for the Ultra. If vLLM, MLX, llama.cpp, or something that doesn't exist yet is obviously better six months from now, I'll change it. The models and backends are replaceable pieces. The architecture is what I actually care about. How it fits together The main desktop with the two 5060 Tis is effectively the home of the environment. That's where OpenChamber runs and where the project workspaces live. Conceptually: Desktop / MacBook / iPhone ↓ OpenChamber / persistent project workspace ↓ Primary agent: M5 Ultra / Qwen ↓ Worker 1: Occamy / ~175K context Worker 2: Occamy / ~175K context ↓ Optional ZDR cloud model when necessary The Mac Studio acts as the primary local inference appliance. The 5060 Ti box provides the high-speed worker capacity. When I'm doing something substantial, the primary agent can remain focused on the overall problem while delegating research, coding, analysis, validation, etc. to the two worker contexts. This is the part of local inference that interests me much more than "I can chat with a model without the internet." I'm trying to build a persistent agentic workspace, not a local chatbot. Remote usage My MacBook and iPhone can connect back into the environment remotely through my self-hosted infrastructure. So whether I'm sitting at my desk, using my laptop elsewhere, or grabbing my phone, I'm interacting with the same underlying workspace instead of creating isolated ChatGPT/Claude/etc. conversations all over the place. The project files and long-term workspace stay home. Inference stays local whenever possible. Privacy model I wouldn't describe this as "nothing ever touches the cloud." That's not true, nor is it really the goal. I'd describe it as: Local-first, private-by-default, cloud by deliberate escalation. The default inference path is hardware I control. My Mac Studio doesn't even have WAN access. But OpenChamber is also configured with a ZDR API provider. If there's a task where a larger model is clearly useful enough to justify leaving the local environment, I have that option. That's an intentional privacy downgrade for that specific request rather than the default state of the system. I think that's a much more realistic privacy model than pretending I'm never going to want access to a model larger than I can physically run. What am I actually betting on? This is probably the most important part. I'm not claiming this is currently cheaper than simply paying for AI subscriptions. That depends enormously on your usage. I'm buying something different. I'm buying a permanent inference floor. If an API provider doubles its price, changes rate limits, retires my favorite model, alters its privacy policy, or disappears, my local capability doesn't change. The two 5060 Tis I already own will continue running whatever they can run. For the next three years, the Mac gives me a known amount of high-memory inference capacity at a fixed monthly cost. And I'm betting that software improvements will continue extracting more capability from that same hardware. A model that requires 150 GB today might eventually have a much smaller model matching it. A model that's painfully slow today might eventually benefit from better quantization, MTP, sparse attention, caching, or a better inference engine. Agent architectures might improve enough that several smaller models working together outperform the way we're currently throwing one enormous model at everything. That's the upside. There is obviously a downside. If cloud inference becomes dirt cheap, open models stagnate, models become massively more memory hungry, or next-generation hardware delivers absurdly better performance per dollar, then this was not the optimal financial move. That's the bet. I'm comfortable taking it over a three-year window. Cost so far My major AI-specific costs are roughly: 2x 5060 Ti: $800 total, purchased outright 96 GB M5 Ultra: $117/month VPS: $5/month on my existing/possibly legacy plan Electricity: definitely not zero :) There's also all the homelab equipment, UPSes, networking, storage, etc., but I don't consider that an honest AI-specific cost because I already ran a homelab before doing any of this. Obviously someone starting from nothing would have a very different cost calculation. Why I'm posting this Mostly because I'm curious how other people are approaching the same problem. I feel like most discussions still fall into one of two buckets: "Just use the API, it's cheaper." or "Look at this giant local model I managed to load." I'm much more interested in the middle layer: How do you build local inference into something that behaves like a persistent service? How do you make it available everywhere? How do you handle multiple agents? How do you isolate the actual inference hosts? Where do you deliberately allow cloud models into the workflow? And at what point does owning predictable inference capacity become more valuable than continually renting access to better models? That's the experiment I'm running.   submitted by   /u/No-Name-Person111 [link]   [comments]