Checks That Pass on Nothing

Why this page exists

Everything we publish about a plant has to cite a named university extension service, and a person has to have read that source. On top of that sits a layer of about forty automated checks that run before anything can publish — the part that catches what a careful read misses.

Over two days, nine of those checks turned out to be passing while examining nothing at all. Not failing quietly. Passing, confidently, on an empty read. Seven of the nine shared a single defect; the other two failed in their own ways.

Alongside them we found two places where no check existed at all, and one new guard that protected nothing. Those are a different problem and are kept separate below rather than folded into the count.

None of them produced a wrong sentence on a page. We found them ourselves, on our own tooling. We are writing them up because a methodology page that lists only the parts that worked is marketing, and because the specific ways they failed are more useful than the fact that they did.

The worst one: a counter that a careless edit cannot disturb

One test is meant to prove that every field of your garden record survives the trip to the server and back — that nothing you write is silently dropped.

To check it, we deleted four columns from the code that sends the data. The test reported seven passes.

It had a safeguard, and the safeguard is the interesting part. It compared the number of columns named in the statement against the number of value slots waiting to receive them, on the theory that a mismatch means somebody has broken something.

But a careless edit removes a column and its slot together. The two numbers stay equal, the safeguard stays quiet, and the invariant survives the exact mistake it was written to catch. It could only ever have caught a very specific and rather unlikely kind of typo — and never the ordinary way this actually goes wrong.

Seven checks that read our own comments as if they were code

Several of our checks work by searching the codebase: is this function actually called anywhere? Does anything still use the old field name?

Seven of them were searching the raw text of the files — which includes the comments. So a comment mentioning a name satisfied a check asking whether that name was used. Writing this used to call `oldThing` in an explanatory note was enough to convince the check that `oldThing` was live and in service.

One of those seven guards a rule this project has broken three separate times: that all date arithmetic lives in one place rather than being reimplemented per screen. It had gone completely silent. The comments explaining the rule were, by themselves, enough to satisfy the check enforcing it.

A test that searched for a digit in a field made of digits

Another test was meant to confirm that a temperature record carries a real sourced figure rather than an empty note. It searched the note for a digit.

Every note on a temperature field quotes a temperature. The test could not fail. It had never failed. It passed on every record, every run, and told us nothing at any point.

Two that were not checks at all, because nothing was watching

We deployed code that referenced database columns which did not exist. Every attempt to record what you had done in the garden would have failed — caught before it reached anybody, but only by a person looking.

Worse, and quieter: a database change from an earlier feature had been sitting unapplied since that feature shipped. Its counter had been writing nothing the entire time. Nothing in the system connected a pending database change to the code that depends on it, so there was no check to fail. The absence was not a broken check; it was a missing one, which is harder to notice because there is no green tick to mistrust.

And one guard built for a problem that was already fixed

Rounding this out, a new guard was written against a failure that had already been solved elsewhere, aimed at the wrong component entirely. It was discovered by testing it — deliberately breaking the thing it was supposed to protect and finding it did not care.

It was reverted rather than shipped. A check that guards nothing still costs something: it takes up the attention that belongs to the checks that work, and it makes the suite look more thorough than it is.

The shape all five share

Three defects across nine checks, two absences, and one decoration — and they are one thing: every one of them was confirming rather than examining.

A broken instrument does not announce itself by failing. It announces itself by answering — promptly, in the expected format, with a green tick. Everything downstream then treats that answer as evidence, and the more checks a project has, the more comfortable the silence becomes.

Note what is not the lesson here. Nobody was careless, and no amount of additional care would have caught any of them. A person reading a passing test sees a passing test. The only thing that distinguishes a working check from a decorative one is watching it fail.

What we do differently

• Break it on purpose, one thing at a time. A new check is not trusted until somebody has deliberately introduced the fault it is meant to catch and watched it go red — and confirmed that the failure it printed names the thing being tested rather than something adjacent.
• Report coverage, not only failures. Several of these were invisible because the output said 0 problems rather than 0 problems across 94 items. A check that says how much it examined cannot hide an empty read.
• Searching source means deciding about comments. Every check that greps our own code now has to state whether prose counts, because the default answer was silently the wrong one seven times.
• A safeguard that both sides of an edit move together is not a safeguard. If the ordinary mistake preserves the invariant, the invariant is measuring something else.

What this still cannot do

No check reads a source and tells you whether we described it fairly. That is a person's job and it stays a person's job — a machine can confirm a quoted sentence appears in the page we credited, and it cannot tell you the page was about a different crop.

And the ones above are the ones we found. We have no way to state how many remain, because a check that passes on nothing looks exactly like a check that passes. The honest version of this page is that we now have a habit rather than a guarantee.