1. A short history of where QA's bottleneck sat
| Era | Bottleneck | Dominant artifact | How it failed |
|---|---|---|---|
| 1990s waterfall | Finding defects at all | Manual test scripts, defect logs | Found late, fixed expensively |
| 2000s automation | Execution cost | Recorded UI suites | Brittle; maintenance exceeded value |
| 2010s shift-left / CI | Feedback latency | Unit suites, line coverage % | Coverage optimized instead of risk |
| 2020s microservices | Cross-service behavior | Contract and integration tests | Coverage looks healthy, incidents cluster elsewhere |
| 2025+ AI codegen | Confidence, not code | Agent-written code and agent-written tests | The author grades its own work |
The through-line: each era solved its predecessor's bottleneck and inherited a new one. Writing software has gotten steadily cheaper. Proving it correct has not. In the AI-codegen era the gap widens for a specific reason, the same system that writes the code is often asked to judge it, and correlated reviewers do not catch correlated mistakes.
2. The experiment
Two instruments were pointed at one production codebase (Twenty CRM, ~24,000 files, an open-source CRM with an active issue tracker).
| Frontier model (Claude Fable 5) | Deterministic graph (TestMojo MCP) | |
|---|---|---|
| Method | Full spec-density census, then deep reads of the highest-scoring gaps | Static graph of every behavior, scored on structure |
| Volume examined | ~75 files, model-selected | 4,849 behaviors, all of them |
| Output | Defect mechanisms with reasoning | Ranked exposure with evidence tiers |
| Repeatability | New sample each session | Byte-identical ranking for a given version and commit |
| Marginal cost | Dollars per session | Cents per run, local |
| Blind spot | Whatever it chose not to read | Semantics: cannot infer a race from shape |
Overlap between the two output lists: near zero.
Each instrument's blind spot is the other's finding.
2b. Second repository, different language
A follow-up run took the same method to a Go distributed workflow server, 8,726 public behaviors, no framework adapter available for its handler-registration style, a codebase and language entirely unlike the first. The top-20 exposure list concentrated on the subsystems that repository was actively rewriting: one operations subsystem alone took 7 of 20 slots, alongside a new execution framework, worker versioning, queue partition scaling, and a scheduler migration path. No roadmap or issue-tracker input was used; the ranking came from call structure, churn, and test linkage.
One repository is a demonstration. Two, in different languages with different architectures, is a method.
The proof ladder also closed end-to-end on that run, and one of the three proofs is self-incriminating in a useful way:
- •Two came from generated tests. The first draft asserted that an error type would echo back the message it was constructed with. The real code hardcodes its string and ignores that field. Executing the test caught the tool's own model inventing an expected value. The oracle was corrected from source; mutation proof then closed both methods.
- •One came from the repository's own test suite. The harness mutated a cancellation state-machine behavior, and an existing test caught it, promoting that link from statically linked to proven. Every attempt that could not establish a clean baseline stayed unclosed.
A verification layer that catches its own generator is the strongest available evidence for the decorrelation argument in §4.
3. What the disagreement revealed
Two agreements and one disagreement carry most of the analytical weight.
Agreed, at the top of one list and high on the other: both instruments independently flagged the message-queue abstraction, 184 importers, a single spec file, as a high-risk untested surface. The model placed it in its top tier; the graph ranked it #18 of 4,849. Different positions, same call.
Agreed, at the bottom: both independently deprioritized the same view/page-layout family. Convergent deprioritization is harder to manufacture than convergent alarm, and is the stronger calibration signal of the two, negatives carry no marketing value, so nobody games them.
Disagreed, productively: the model's top finding (a JWT key-rotation race) sat in the graph's data at unconfirmed-candidate tier but outside its top twenty. Tracing the gap isolated two scoring factors, silent-failure mode and audience sensitivity, documented in the graph's own methodology but not yet weighted in its shipped formula. The disagreement functioned as instrument calibration and produced a scheduled fix with a pre-written acceptance test.
That is the useful pattern for practitioners: when two decorrelated instruments disagree, the delta is a measurement, not noise.
4. Properties that do not arrive with better models
| Property | Why it matters operationally | Why sampling cannot supply it |
|---|---|---|
| Determinism | A merge gate needs a number that does not drift | Sampling distributions differ by construction |
| Exhaustiveness | No blind spots; every behavior scored | Context windows force selection |
| Executed proof | "This test fails when the behavior is mutated" is evidence | Reasoning predicts; it does not observe |
| Accumulated state | Trends, deltas, a proof ledger across runs | Each session starts empty |
| Auditable inputs | Provenance line per run: source root, commit, history completeness, churn window, version, input hash | Prompt and sampling state are not reproducible |
| Decorrelation | The verifier must not be the author | The author cannot audit itself |
The last row is the load-bearing one. As more code is written and reviewed by models, the enterprise question shifts from "is this correct?" to "who says so, and are they independent?" Auditors are not employed by the companies they audit, for reasons that have nothing to do with capability.
A precision worth stating: Two layers behave differently. The map is deterministic, same version, same commit → byte-identical behaviors, tiers, scores, ranks. Generated tests are model output and vary between runs. They are drafts, not deliverables: each is compiled against the target repository before being presented as runnable, and failures are demoted to written intents carrying the compiler's own diagnostic. Nothing reaches covered without execution.
5. Evidence tiers instead of a coverage percentage
Most tools report one number. This one refuses to, and reports a distribution.
The claim is not "92% of this codebase is untested." It is "for 92% of behaviors, the only linkage discoverable is a name resemblance." That distinction is the entire epistemology, and it is unusually disciplined for a vendor metric, the larger, vaguer number was available and not taken.
6. Where this sits in a delivery pipeline
Reasoning proposes, only execution promotes.
Two rules make the loop trustworthy rather than decorative: reasoning proposes, only execution promotes, and nothing is labeled covered on the strength of a name match. Evidence stays tiered, mutation-proven, statically linked, unconfirmed candidate, no signal, so a lead is never quietly counted as a guarantee.
7. What this implies for the QA profession
The role is not disappearing; its center of gravity is moving from producing tests to adjudicating them.
| Phase | Window | What the QA engineer does | What is scarce |
|---|---|---|---|
| Tool-augmented tester | 2024–2025 | Uses copilots to draft cases | Prompt quality |
| Agentic orchestrator | 2025–2027 | Designs multi-agent pipelines; curates their inputs | Pipeline design, taxonomies |
| Quality strategist | 2027–2030 | Sets standards, tunes risk models, judges whether generated tests catch anything | Judgment, domain depth |
Three capabilities compound across all three phases: enough engineering to read a diff and map it to behavior; enough prompt and skill authoring to encode institutional QA knowledge in a form agents follow repeatably; and enough data curation to keep requirements, taxonomies, and golden examples clean, because generation quality tracks input quality more tightly here than anywhere else.
The practical implication for teams: invest in the classification and adjudication layer, not only in more test scripts. Suites scale with headcount. Taxonomies, gates, and evidence ledgers scale with the codebase.
8. Honest limits
An assessment that only lists strengths is marketing. Five caveats belong on the record.
The static-analysis layer is replicable.
A capable model improvised an equivalent census in a handful of shell commands. The defensible parts are narrower: determinism, executed proof, accumulated evidence, and the discipline that refuses to overstate.
Proof coverage is early.
Three mutation-proven behaviors on an 8,726-behavior server, and four of eighty-one on a small library. The mechanism now demonstrably works end-to-end on a large repository, including one proof minted by the repository's own existing test, but three of 8,726 is a rounding error. "Executed proof" is a demonstrated capability at demonstrated small scale; the ladder is built, not populated.
Static linking is conservative by design.
On repositories whose tests import through workspace aliases, module-path dependencies, or same-package conventions, the tool reports fewer links than exist, and reports that gap rather than guessing. Under-counting is the intended failure mode; it is still a gap, and it inflates the candidate tier.
Map quality varies by stack.
The Go run reported no recognized framework, because no adapter exists yet for its handler-registration style. The consequence is visible in the output: high fan-in utility functions surface near the top, because structural weight is all the ranker has to work with. Framework adapters look like the gating item for broader applicability.
Benchmark contamination is real, and partly addressed.
The original reference repository was used to find and fix the scorer's own defects, which makes it a demonstration rather than a test. The Go server run was genuinely out-of-sample, a different language, a codebase the tool had never seen, and an incident history the evaluator could check independently. That is n=1 out-of-sample: a start, not a track record.
9. Verdict
On semantic depth, frontier models win now and will win by more later. Any product positioned against them on that axis is positioned against the tide.
The defensible position is adjacent, not opposed: a cheap, deterministic, locally-run adjudicator that enumerates exposure, routes expensive attention to the top of its own list, and refuses to promote anything without execution. It gets better as models improve, because it consumes them; and it gets more necessary as models improve, because more of the code under review was written by something that cannot grade itself.
One instrument reads code and produces defects. The other enumerates exposure and produces a gate. Mature teams will run both. Most teams currently run neither.
Method note: figures come from a head-to-head run on one public TypeScript repository (Twenty CRM, 4,849 behaviors), plus runs on four further public repositories of differing shape, a Go distributed workflow server (8,726 behaviors, where the three mutation proofs were minted), a multi-program Go samples repository, a TypeScript monorepo SDK, and a CommonJS Node web framework. All reports are reproducible from the published tool with one command.
Verify it yourself
Clone twentyhq/twenty, run the command above, and open .orangepro/behavior-coverage.html when it finishes.
Open source. MIT licensed. No account required. Results in under 5 minutes.
Want to see it on your codebase?
We'll run TestMojo against one of your repos and walk through the report together. No commitment.