I Built a RAG Agent That Was Always Confident and Often Wrong. Here's What Fixed It.A few months into running one of our newly deployed project - an internal AI agent that let our support team ask plain-English questions about customer accounts instead of digging through four different admin panels, I got a Slack message that started with "hey this is probably nothing but."It was not nothing. The agent had told a rep that a customer's subscription had been cancelled. It hadn't. The cancellation request was sitting in a queue, not yet processed, and our agent had pulled from a table that only updated once a batch job ran every few hours. The rep, trusting the tool we'd told her to trust, had already told the customer their service would end at the end of the month.Nobody yelled at me, which honestly made it worse. Everyone was very understanding. That's usually a sign you're about to spend your weekend in the data layer instead of doing literally anything else.Here's the thing that took me an embarrassingly long time to internalize: I had spent most of my effort on the model side. Prompt structure, retrieval relevance, chunk sizes, which embedding model gave better recall on our knowledge base. All of that mattered, and all of that worked fine. The retrieval was relevant. The generation was fluent. The agent answered exactly the question it was asked, using exactly the data it had access to. The data was just wrong, and the agent had absolutely no way of knowing that, because nothing in the pipeline ever asked it to check.That's the part nobody really warns you about when you're building one of these things. An LLM doesn't have a gut feeling that a number looks stale. A human support rep glancing at an admin panel might pause and think "huh, that field hasn't moved in a while, let me double check." The agent has no equivalent instinct. It gets a row from a table, treats that row as ground truth, and writes you a confident paragraph about it. Garbage in, fluent garbage out, and fluent garbage is so much more dangerous than obvious garbage because nobody questions it.I went looking afterward for whether this was just us being sloppy or an actual pattern, and it's very much a pattern. Atlan published research on agent failures in production that found the majority of incidents they looked at traced back to the data layer feeding the agent, not the model itself, and that those failures are uniquely hard to catch because they don't look like failures, they look like answers. Which tracks exactly with what happened to us. Nothing crashed. Nothing threw an error. The agent just confidently said something false.So here's roughly what we changed, in case it's useful to anyone building something similar.The first fix was embarrassingly simple and probably should have been there from day one: every piece of retrieved data now carries a timestamp of when it was last synced, and the agent's system prompt explicitly tells it to surface that timestamp when the data relates to anything time-sensitive, like subscription status, payment state, or account standing. Something like:retrieved_record ={ "subscription_status": "active", "last_synced": "2026-04-12T03:00:00Z", "source": "billing_db_replica"}If that timestamp is older than some threshold for that specific field (we landed on fifteen minutes for billing-adjacent data, a lot looser for things like account history that don't change moment to moment), the agent is instructed to flag it rather than state it flatly. "Subscription status as of the last sync was active, but this was [X minutes/hours] ago, you may want to confirm directly" is a very different sentence than "the subscription is active," and it's the difference between a rep double-checking and a rep promising something to a customer.The second fix was less about the agent and more about the plumbing underneath it. We'd been treating the replica database as good enough because it was "close enough to real time," without ever actually defining what close enough meant for each kind of data. Turns out nobody had. We went through and explicitly classified data by how stale it's allowed to get before the agent has to either refuse to answer confidently or fall back to a live query instead of the cached version. Billing and account status: near zero tolerance, query live or flag it. Historical support tickets: a few hours of staleness genuinely doesn't matter. Treating all of it the same was the actual root cause, more than any one bad sync.The third thing, and probably the one that mattered most long term, was just building a dashboard nobody had to remember to check, that flagged when any data source the agent depended on had gone stale past its threshold. Before, staleness was invisible until it caused a visible problem, like a rep telling a customer the wrong thing. Now it shows up as a yellow flag in a Slack channel before anyone downstream acts on it.None of this required touching the model at all. We didn't switch providers, didn't change the prompt structure beyond adding the staleness instruction, didn't retrain anything. The fix lived entirely in the boring layer underneath the part everyone gets excited about.I think this is worth saying plainly because of where the industry's attention currently sits: a huge amount of energy right now is going into agent frameworks, tool-calling patterns, multi-step planning, which model reasons best. All of that is genuinely interesting work and I don't want to undersell it. But none of it helps you if the thing the agent is reasoning over is wrong, and "wrong" in a way that looks exactly like "right" until someone downstream gets burned by it. We didn't have a model problem. We had a "nobody had ever explicitly decided how fresh this data needs to be before an autonomous thing acts on it" problem, and that's a much less glamorous bug to go looking for.If you're building something similar, the question I'd actually sit with isn't "is my retrieval good" or "is my model good." Both of those are checkable, and you'll check them, because they're the fun part. The question that's easy to skip is: for every piece of data this thing can act on, what's the actual tolerance for staleness, who decided that, and what happens when a number crosses that line — does the agent know, or does it just keep talking?We found out the answer to that the hard way. Cheaper to find out from a blog post than from a Slack message that starts with "hey this is probably nothing but."\