A test that cannot fail is not a test

As Jiva grew I got scared of silently breaking old things. So I built a regression net — and then made myself prove it could actually catch a mistake.

As Jiva grew, I got scared of a specific thing: fixing one part and silently breaking another, and not noticing until a user did. So I built a regression harness — feed a fixed set of made-up meetings through the whole processing pipeline, and compare every output against a saved "golden" copy. If an output changes unexpectedly, that is a regression and the build goes red. If the change was intended, I re-bless the golden copy on purpose.

That much is standard. The part I want to write about is a rule I made myself follow: every test has to be able to fail. It sounds obvious and it is constantly violated. A test that always passes tells you nothing — it is a green light wired to stay on — and the danger is that it looks like coverage while protecting nothing.

So I mutation-test the tests. I deliberately introduce a small wrong change — a price in a fixture from 40 to 45 — and confirm the harness actually goes red, in exactly the files it should: the decision list, that meeting's summary, the index. If a broken input does not turn the test red, the test is decoration, and I would rather know that than trust it. Note that the primary driver here is not coverage in the abstract, rather earned confidence: a test is only worth the trust I put in it if I have watched it catch a real mistake at least once.

The general principle sits well beyond code. A check you have never seen fail is not a safety net, it is a comfort. This is the same lesson as the backup script that congratulated itself while backing up nothing, and the download bar that looked identical whether it was working or dead: the dangerous state is a reassuring signal you have never verified. So verify it. Break it on purpose, once, and watch it complain.

More soon.