Your Health Checks Are Verifying the Wrong Thing

Wait 5 sec.

Four bugs in one week. All of them passed monitoring. All of them were the same bug.Last week I found four separate faults in a system I built. A financial figure that was wrong by 16 times. A metric frozen for a month while appearing to update. Nine decommissioned processes still holding live credentials. And 149 backup files serving secrets over HTTP to anyone who asked.Every one of them passed monitoring the whole time. Not because monitoring was broken. Because every check was verifying that something existed rather than that it was true.That distinction is the whole article. If you run scheduled jobs, agents, or anything that reports its own health, you have written this bug. I can almost promise it.The first one, told properlyWe report monthly recurring revenue on an internal dashboard. It showed a number. The number was wrong by a factor of sixteen.It had been wrong for weeks. Every automated health check passed the entire time.The check did what it was written to do: read the value, confirm it is not null, pass. The value existed. The value was well-formed. The value was a number. The check went green.Nothing anywhere asked whether the number matched the contracts it was supposedly derived from. Reconciling against source data is real work. You need a second source, a join, and a tolerance. Asserting non-null is one line. So we wrote the one-liner, watched it go green for weeks, and believed it.A dashboard that is green for the wrong reason is worse than one that is red. Red prompts investigation. Green manufactures confidence in a number nobody has checked.Then it happened three more timesOnce I knew what shape I was looking for, I found it everywhere.The frozen metric. A weekly job records our domain authority. It ran every week without fail. The monitor confirmed a fresh row every week. But when the upstream API failed, the job wrote the previous value forward rather than writing nothing. So the row was always fresh, always present, always passing. The underlying number had not been genuinely measured in four weeks. The check verified that a row existed. It could not tell a measurement from a copy.The retirement that retired nothing. We decommission agents by setting status to retired. Nine of them were retired. All nine still had working access. The reason is that credentials were never scoped per agent in the first place. Everything executing shared global secrets by category. So retirement changed a label in a database and left the door code unchanged. The check verified the status field had changed. Nobody had asked whether the status field controlled anything.The secrets scan that scanned the wrong files. After rotating a leaked API key, we scanned for the old key and found nothing. Clean. Except the scan pattern only matched .php files, and the key was sitting in 149 .bak files, several of them world-readable and, worse, served over HTTP by the web server. The check verified that no matching file contained the secret. The pattern defined what counted as a file.Four failures. Four passing checks. One shape.Why this happens to everyoneExistence is cheap to verify. Truth is expensive.To confirm a value exists, you read it. To confirm it is correct, you need an independent second source, a reconciliation rule, and a decision about tolerance. That is an afternoon of work per metric rather than a line per metric.So the cheap check gets written first, ships, goes green, and is never revisited. Nothing prompts you to revisit it, because it is green.The failure mode compounds in a specific way. The more monitoring you add, the more confident you become, without necessarily becoming more correct. We had 51 active agents and a self-check suite passing 12 out of 12. That suite was thorough about structure and silent about truth.The fix, as a patternFour rules, all cheap to implement, that would have caught every one of the above.Every metric carries its source and the time it was last genuinely verified. Not when it was last written. When it was last measured. A value without provenance renders as unverified rather than as a number. This sounds pedantic until a stale value silently sets a target for you.A failed fetch writes nothing. Never carry a value forward. Never stub. If the upstream call fails, the metric ages past its freshness window and starts reporting itself as unverified. Degrade loudly. The carry-forward stub is the single most dangerous line of code in a monitoring system, because it makes failure look identical to success.Green must mean reconciled against something independent. If a check can pass using only the data it is checking, it is not a check. It is a formatting assertion. The financial number needed reconciling against the contracts. The domain rating needed a freshness window. The retirement needed a live credential test.Test the mechanism, not the record. This is the one people skip. Marked as revoked is not revoked. If you cannot answer who kills this access, how fast, and have we ever actually tried it, then you do not have revocation. You have a plan for revocation, and a database row that describes it. Retire a throwaway agent, then attempt a real call with its credential. Time it. If the call succeeds, your retirement is decoration.The corollary worth sitting withA system that cannot be wrong and loud will be wrong and quiet.Every one of these faults existed for weeks. None of them were new. What changed was that something finally checked the right thing, and then they all surfaced at once, which feels like the system falling apart and is actually the opposite.If your monitoring never delivers bad news, that is not evidence that nothing is wrong. It is evidence that nothing is checking.Go and run this todayThree questions you can answer about your own system in about ten minutes.Pick your most important number. What independent source would you reconcile it against? If the answer is none, you are asserting existence.Find a job that pulls from an external API. Read the failure branch. Does it write anything on failure? If yes, you have a carry-forward bug waiting.Take one thing you decommissioned this year. Find the credential it used. Try it. If it still works, your retirement is a label.I found four bugs by asking these. I expect you will find at least one.