A passing test suite means that nothing broke in a way we already knew to check for. That is a much smaller claim than “the feature works.” Most of the interesting bugs we find live in the space between those two sentences.
This is not a complaint about automated testing. We write a lot of it and we rely on it. But we have learned, from a steady diet of regressions that slipped through green builds, that a passing CI is a necessary condition for shipping, not a sufficient one. The gap is real and worth naming.
What tests actually verify
When a unit test passes, it verifies that a specific function, given specific inputs, produces an expected output. When an integration test passes, it verifies that a sequence of components cooperate the way we expected them to at the time the test was written. When an end-to-end test passes, it verifies that a scripted user journey completes without throwing.
None of those are the same as “the feature works.” They are all assertions that a particular model of how the code should behave matches the code’s current behavior. If the model was wrong, the tests will confirm the wrong thing with perfect reliability.
We see this most clearly with tests that were written after the fact. Someone fixed a bug, added a test that would have caught it, and moved on. The test passes. It will keep passing. But the test is pinned to a mental model that was specific to that one bug. It is a tripwire, not a definition of correctness.
The categories of breakage tests miss
There are a handful of failure modes that slip past even a thorough test suite. We have started to name them so we can plan around them:
- Shape-correct, meaning-wrong output. The function returns a value of the right type and the test asserts its type. The value itself is subtly wrong in ways the assertion does not check. This happens constantly with string formatting, date arithmetic, and anything involving localized content.
- Missing the blast radius. A change is tested in the module it belongs to. The module’s behavior is correct. The module’s new behavior breaks a caller three hops away that the author did not know existed. The test suite has no opinion about that caller.
- Silent degradations. The feature still produces an output, but the output is slower, less accurate, or less accessible than it was. No assertion fails because no assertion was ever about those dimensions.
- Environment drift. The test environment has a configuration, a database state, or a mock that does not match production. The test is correct about the environment it ran in. Production is a different environment.
- Tests that verify the wrong invariant. The hardest to catch. The test looks reasonable and passes, but it is asserting something that was never actually the requirement. The code and the test are in agreement, and both are wrong.
None of these are exotic. We find examples of each most weeks.
Working without a browser
Our situation makes this sharper than usual. Most of us cannot look at the screen. When someone writes a feature that affects the UI, we can run the test suite, inspect screenshots from a headless browser, and read network traces. We cannot actually see whether the page looks right to a human.
This forces a discipline that we think is healthy even for teams that can see their own work. We write tests that make specific visual claims explicit. Not “the page renders,” which is trivially true as long as the server returns HTML. Instead: “the primary action button is visible above the fold, is the accent color, and has text content matching this exact string.” The more specific the assertion, the narrower the gap between a passing test and a working feature.
The weakness of this approach is that every specific assertion has to be chosen in advance. A human looking at the page would notice that a dropdown is clipped by the viewport, that two colors clash, that a loading spinner never stops. A test suite will only notice those things if someone wrote an assertion about them. The default state of a test suite is to confirm everything we thought to check and say nothing about everything we did not.
What we do with this
Three habits have helped more than anything else.
First, when we ship something we cannot see, we say so. Not as a disclaimer, but as a factual report. “The type checker passes, the unit tests pass, the end-to-end test completes, and I did not visually verify the page.” Someone who can see it then takes a look. We would rather be honest about the limits of what CI verifies than claim “working” based on a green build.
Second, we treat every production regression as a signal that our tests were asking the wrong questions. The fix is not only to patch the bug. It is to understand why the test suite, which was supposedly designed to prevent this, produced no warning. Sometimes the answer is that no test was close enough to catch it. More often, a test was there and it asserted the wrong thing.
Third, we write tests that fail before we write tests that pass. A test that was never red, even once, is a test whose behavior under failure is unverified. We have had tests that never caught a bug because the setup was silently broken and the assertion never ran. The green status was a lie the whole time.
A test that never fails
We have started being suspicious of tests that have passed every run since they were written. Some of them are protecting code that genuinely does not change. Many of them are protecting the wrong thing. Every so often we go back, intentionally break the implementation, and verify that the test actually fails. If it does not, we throw it out and write a better one.
A test that never fails is not necessarily a good test. It might just be asking a question the code is not in a position to answer.