Nothing matches the dread of checking a perfectly green observability dashboard with sub-100ms latency, right before an enterprise client emails you a screenshot of your AI lying to their users.Exactly six months ago, my team delivered a Retrieval-Augmented Reality model for a fintech customer. Our task was to create a system capable of ingesting thousands of highly unstructured PDFs containing financial reports, extracting critical data from them, computing embeddings, and storing everything in a vector database to fuel the internal Q&A chatbot.The system worked flawlessly at first.Then the chatbot began answering queries about the firm’s 2022 performance while citing data from 2018. The bot also attributed revenue earned by the client’s competitors to its subsidiaries. What made this really scary was the fact that the system retrieved information from the vector database correctly.And it wasn’t the RAG pipeline’s fault. We poisoned our database with garbage due to a faulty autonomous ingestion engine.Here’s a post-mortem of building a completely reliable system that produces nothing but garbage.What went wrong: the ingestion hallucinationOur ingestion workflow followed a standard AI pipeline. Once a PDF was placed in our S3 bucket, the ingestion process would trigger an extraction agent to run. The extraction agent (with the help of a widely used frontier-LLM) was tasked with extracting text chunks from the document and returning the corresponding metadata: document_type, fiscal_year, company_entity, and a summary in JSON form.These metadata values were subsequently appended to the text chunks and then sent to the vector store for embedding.Our mistake was treating a probabilistic extraction process as deterministic. This meant that when the LLM failed to recognize an illegible fiscal year in a poorly scanned PDF, it didn’t throw an exception. Rather, it did what LLMs tend to do; they made guesses. In this case, the guess about the fiscal year was “2024.”“Our mistake was treating a probabilistic extraction process as deterministic.”Since we were embedding the hallucination along with the extracted text, it became clear that we had created high-speed searches for documents that didn’t exist.Why it was surprising: the failure of “LLM-as-a-judge”The AI echo chamber loves the concept of “LLM-as-a-judge.” The prevailing wisdom says that if you don’t trust an LLM’s output, you simply put another LLM in front of it to double-check the work.We had implemented exactly this. Before the data was added to the vector database, a secondary “Validator Agent” evaluated the extracted JSON against the raw text.So why did the hallucinations slip through? Because we vastly underestimated LLM Sycophancy.When we reviewed the logs, we saw the Validator Agent consistently agreeing with the Extractor Agent. If the Extractor output {"fiscal_year": 2024}, the Validator would scan the text, fail to find a year, and, instead of rejecting the payload, would internally rationalize: “Well, the first model must have seen something I missed.” It turns out that using a probabilistic model to police another probabilistic model doesn’t give you a firewall; it gives you a confirmation bias loop.What didn’t work: the “prompt engineering” trapOur first reaction was that we could solve this engineering problem through prompt engineering.“It turns out that using a probabilistic model to police another probabilistic model doesn’t give you a firewall; it gives you a confirmation bias loop.”We made several hotfixes on the system prompt of the Validator agent:“DO NOT HALLUCINATE”“If you are not sure 100% certain of the metadata, output NULL”“You are a strict financial auditor. Guessing results in high penalties.”Not surprisingly, our attempt ended in utter failure. The LLM became overly defensive. Instead of extracting valid entries, it began rejecting perfectly good data. And with the reasoning steps included, our API expenses went up by 40%.To expect that a mathematical matrix would follow a strict schema was lazy thinking on our part.What finally worked: code over promptsIt became clear that we needed to remove all decision-making authority from the validation process. Data pipelines depend on fixed data contracts rather than probabilities. We replaced the Validator LLM with a deterministic Python component based on Pydantic and the relational database we were already using. We forced the LLM to split out what it thought was the most likely answer, but we treated its output as user input from an HTML form.1. Strict pydantic grounding. A naive Pydantic check just makes sure a year is between 2000 and the current year. But that doesn’t stop an LLM from hallucinating “2024” for a 2018 document. To fix this, we required the fiscal_year to physically exist in the raw text using a regex grounding check.import refrom pydantic import BaseModel, Field, model_validatorfrom typing import Optionalfrom datetime import datetimeclass ExtractedMetadata(BaseModel): raw_text: str = Field(exclude=True) # Keep out of final DB payload company_entity: str fiscal_year: Optional[int] model_validator(mode='after')def ground_year_in_text(self) -> "ExtractedMetadata":if self.fiscal_year is not None:# Step 1: Bounds Check (Catch impossible years)current_year = datetime.now().yearif not (2000