We use an LLM agent to pull structured data from web pages. To keep cost down, the agent turns what it learned on a site into reusable rules that run later without a model call, and the model only steps in when the rules fail. The rules look right on the pages they were learned from. The hard part is what happens after: new pages, no ground truth, and a rule that still returns something that looks fine but is wrong. How do you catch "plausible but wrong" once such rules are live? Sampling with a model as a second opinion, canary pages, cross-checking sources, something else? Which drift signals have worked for you in practice (page structure changes, fill rates dropping, value ranges), and how often did they fire for no reason? Interested in what people actually run, not what the vendors say.   submitted by   /u/LostLeek1304 [link]   [comments]