Why Your Test Coverage Number Is Lying to You
Your SonarQube dashboard says 74% coverage. Your CI pipeline is green. Your team ships on Friday.
Then production breaks in a way no test caught.
Sound familiar?
The coverage number is not wrong. It is just answering a different question than the one you think you are asking.
What static analysis actually tells you
Static analysis tools do one thing well: they scan your source code for known patterns. They find code smells, detect duplication, flag complexity, and measure which lines were executed during your test run.
What they cannot tell you is whether the right things are being tested, whether tests are in the right layer of the testing pyramid, or whether the tests you have would actually catch a regression in the behaviour that matters most to your business.
A test that calls an API endpoint and checks that it returns HTTP 200 counts toward your coverage number. It does not tell you whether the data was saved correctly, whether the right user has access, or whether the downstream system received what it expected.
Test reporting tools have the same blind spot from the other direction. They tell you what ran and whether it passed. They do not tell you whether what ran was worth running.
The gap between "tests exist" and "the system is protected" is where most production incidents live.
The testing pyramid problem in brownfield codebases
The testing pyramid is a well-understood concept: unit tests at the base, integration tests in the middle, end-to-end tests at the top. Fast, isolated, and numerous at the bottom. Slow, brittle, and few at the top.
In greenfield projects, teams start with this in mind.
In brownfield codebases—the systems that have been running in production for years, built by teams that have changed, under deadlines that compressed best intentions—the pyramid is usually inverted or collapsed.
End-to-end tests carry the weight because they were the fastest way to add coverage when someone needed confidence before a release. Unit tests are thin or absent in the modules that matter most. Integration tests exist for the happy path but not for the edge cases that actually fail in production.
The result is a test suite that is expensive to maintain, slow to run, and less protective than its coverage number suggests.
Untangling this is not a one-sprint task. It requires a structured approach: understand what you have, understand what you need, close the gap systematically, and keep it closed sprint by sprint.
The four questions every brownfield team needs to answer
Before writing a single new test, any team working on a legacy or mature codebase needs clear answers to four questions.
Where are the integration boundaries?
Every system has connection points where it talks to external services, databases, authentication providers, message queues, and downstream consumers. These boundaries are where failures propagate silently. A test that never exercises one of these connection points does not protect the behaviour that crosses it.
Where is coverage genuinely thin?
Not just which files have low line coverage, but which modules are uncovered and also high-risk. A zero-coverage utility class is a different problem from a zero-coverage module that sits on a critical integration boundary. Risk-weighted coverage tells a very different story from raw coverage numbers.
What does the code actually do today?
Not what the documentation says it does. Not what the original architect intended. What it actually does when it runs, including the edge cases, the silent failures, and the places where the code has drifted from its original contract over years of changes. Characterization testing surfaces exactly this: the current behaviour, including the deviations, before anyone attempts to change it.
What is the most efficient path to closing the gaps?
Not all gaps are equal. A critical gap in the authentication boundary is a different priority from a low-risk gap in a utility function. The team needs a plan that sequences work by risk and effort, not just by what is easiest to automate.
Why AI changes the economics of gap closure
Historically, the answer to "our test coverage is insufficient" was to allocate engineering time to write more tests. The problem with that answer is that writing tests for a legacy codebase is slow, expensive, and competes directly with feature delivery.
Characterizing an unfamiliar module requires reading code you did not write, understanding its dependencies, identifying its edge cases, and then writing tests that accurately describe its behaviour. All of that happens before you can even begin to close a gap.
AI assistance changes these economics materially. The analysis that previously took a senior engineer days to produce can be generated in minutes when the right context is assembled and the right questions are asked. The test code that previously took hours per module can be drafted in a fraction of that time, with the engineer's role shifting from writing code from scratch to reviewing, configuring, and validating what was generated.
This does not mean AI replaces engineering judgment. It means engineering judgment gets applied to the right things: deciding what matters, validating that generated tests are correct, and ensuring that fixes do not break dependent code that has quietly adapted to existing behaviour.
The result is that gap closure becomes economically viable alongside feature delivery rather than competing with it.
The sprint cadence that keeps gaps closed
Closing existing gaps is half the problem. The other half is making sure new gaps do not form as the codebase continues to evolve.
Every sprint, new modules are touched, new integrations are added, and new behaviours are introduced. Without a check at the start of each sprint, a team can spend months closing last quarter's gaps while silently introducing next quarter's incidents.
A sprint-level cadence that checks the impact of incoming changes before stories begin, re-measures coverage once tests land, and plans the next priority batch transforms gap closure from a one-off project into an ongoing practice.
The coverage change from sprint to sprint gives leadership a meaningful signal: not a static number on a dashboard, but a direction of travel. Are gaps closing faster than they are being introduced? Is the high-risk surface area shrinking? Is the team on track to reach the quality target they agreed on?
These are the questions that turn a test quality programme into a business conversation.
What good looks like
A mature test quality programme for a brownfield codebase produces a few clear outcomes.
The CI pipeline runs faster because tests are in the right layer. Unit tests that were living inside end-to-end specs run in milliseconds instead of minutes. The suite becomes a faster feedback loop rather than a slower one.
Silent failures become visible. The authentication boundary has a test. The data export has a test. The concurrent write path has a characterization test that will catch a race condition if anyone changes it. The team knows what is protected and what is not, and that picture improves sprint by sprint.
Leadership has evidence rather than assertions. A traceability view shows which business behaviours are covered, which are partially covered, and which have no automated protection. A gap closure plan shows how many sprints it will take to reach the agreed quality target at the agreed capacity. A sprint delta report shows whether the team is on track.
The team owns the programme. They run it themselves, without a consultant, using tools they already have. The knowledge of where the gaps are and how to close them lives in the codebase, in the reports, and in the team, not in a document that goes stale.
A different kind of quality conversation
Test quality has traditionally been a technical conversation: coverage percentages, test counts, CI pass rates. These are useful signals, but they rarely translate into the business conversations that determine how engineering time is allocated.
A team that can say "we have 53 requirements with no automated protection, 12 of which are in the authentication boundary, and we have a plan to close all critical gaps in 7 sprints at 40% capacity" is having a very different conversation from a team that says "our coverage is at 74% and we are working on improving it."
The first conversation gives a Product Owner and Engineering Manager what they need to make a decision. The second gives them a number with no clear path forward.
Turning technical findings into that first kind of conversation, with evidence, a plan, and a measurable trajectory, is what a structured test gap intelligence and remediation programme makes possible.
ToolTwist builds software development tools and programmes for enterprise engineering teams. ClarityMap is ToolTwist's Test Gap Intelligence & Remediation Programme for brownfield software teams.
To learn more about ClarityMap, visit tooltwist.com.