Before accepting an AI-generated unit test, ask which mistake it would detect. For important decision logic, make a small, deliberate regression in an isolated copy and check that the relevant assertion fails. Restore the intended implementation and confirm that the test passes again. A green test suite is useful evidence; knowing why a particular test would turn red makes its contribution easier to assess.
This guide proposes a review method for developers and engineering leads using AI to draft tests. It combines explicit expected behavior with a narrowly scoped check of defect detection. The worked example is fictional, and no model-generated tests or mutation-testing products were run for this article.
Coverage identifies execution, not the meaning of an assertion
An assistant can produce a plausible test name, realistic fixtures and an assertion that checks very little. A test called “does not remind closed customers” might only check that the reminder function returns a value. It could pass even when the function returns the wrong decision.
The Google Testing Blog’s 2020 coverage guidance makes the underlying distinction: coverage records executed code, while covered behavior can still be inadequately checked. Its practitioners recommend using coverage alongside other evidence and discuss mutation testing as a way to examine assertions. This predates modern coding assistants; it is an established testing problem that also applies to their output.
A mutation is a small change to the implementation, such as reversing a comparison. Running the tests against that change asks whether they notice it. The change is artificial, so detecting it does not prove that the tests detect every real defect. It provides a specific challenge to a specific test claim.
Research supports inspecting generated tests through separate gates
The 2023 MuTAP paper by Dakhel and colleagues investigated feeding surviving mutations back into prompts to improve generated tests. Its experiments involved Python tasks and models including Codex and Llama 2. The authors identify artificial mutations and the Python datasets as limits on interpreting their results. This is historical research into a feedback method, rather than an estimate of today’s assistants’ reliability.
An industrial account illustrates a different set of checks. In the 2024 TestGen-LLM paper, Meta researchers describe extending existing Kotlin tests and filtering candidates for successful builds, repeated passing execution and increased coverage. These are company-reported results from Meta’s environment. They do not independently validate current assistants, and coverage improvement alone does not establish the correctness of a business requirement.
Our editorial recommendation is to keep those questions separate: does the test run reliably, does its expected result represent the requirement, and does it notice a relevant violation? A passing build answers only part of that review.
Define the result before generating the test
Consider a fictional service that decides whether to send an overdue reminder. Its agreed rule is to send only when the record is open, the current time is strictly later than the due time, and notifications are not muted. At the exact due time, it should return false.
That last boundary is a product decision. Another service could legitimately use “at or after.” The reviewer needs the agreed answer before assessing either implementation or generated assertion. An assistant should not settle an unresolved requirement by copying whatever the current function happens to do.
Testing terminology calls the mechanism for deciding the expected result an oracle. Here, it is the explicit rule and its examples. Record those examples separately from the generated code so that a later rewrite of the implementation cannot quietly redefine what counts as correct.
Ask the assistant to draft tests using the repository’s existing framework, fixtures and conventions. Supply the rule, the boundary examples and the permitted scope. If the input contract does not explain missing due dates or unknown record states, identify that ambiguity for a decision. Do not accept an invented default as though it were approved behavior.
Challenge one condition at a time
The following is a proposed review table for the fictional reminder rule. The expected results come from that rule; they are not measured outcomes from an experiment.
| Deliberate regression | Input that should expose it | Required assertion on intended behavior |
|---|---|---|
| Remove the open-record condition | Closed record, overdue, notifications unmuted | Reminder decision is false |
| Change “strictly later” to “at or later” | Open record, exactly at due time, unmuted | Reminder decision is false |
| Ignore the muted setting | Open record, overdue, notifications muted | Reminder decision is false |
| Always return false | Open record, overdue, notifications unmuted | Reminder decision is true |
The positive case matters. A suite consisting entirely of “do not send” cases could approve an implementation that never sends a reminder. Each row holds the other conditions steady so the intended distinction remains visible.
Use a fixed current time and explicit timestamps in these cases. Avoid relying on the wall clock advancing during the test. Keep external dependencies controlled, while ensuring the assertion inspects the actual decision under review. If a mock supplies the decision itself, the test may merely confirm that the mock returned its configured answer.
For an actual defect with a known reproduction, first consider running the regression test against the faulty revision and the proposed fix. That is more directly tied to the reported failure than an invented mutation. This guide’s table is useful when reviewing newly generated tests for a rule, or when no historical faulty revision is available.
Inspect the failure before recording success
Run the proposed test on the intended implementation first. If it fails, establish whether the requirement, implementation, fixture or assertion is wrong. Changing the expected value until the run turns green would discard the independent reference that makes the test useful.
Next, introduce one selected regression in a disposable working copy with no production credentials or deployment actions. Run the relevant tests and inspect the output. For the exact-deadline mutation, the useful failure would identify the decision as true when false was expected. An import failure or missing fixture does not demonstrate that the deadline assertion detects the changed comparison.
Restore the intended implementation, confirm the diff contains no deliberate regression, and rerun the affected tests. Keep the diagnostic mutation out of the submitted production change. Record what actually happened, including a surviving change or an inconclusive failure, rather than converting every unsuccessful run into a test-quality success.
If an automated mutation tool is already available, read its individual outcomes. Stryker’s documentation distinguishes killed, surviving and uncovered mutations from errors; it also counts timeouts as detected in its score. That accounting convention is useful to understand, but a timeout does not explain whether the intended business assertion worked. Inspect the result behind the percentage.
Keep the review proportional to the risk
Start with changed decision logic where a wrong result has a clear consequence: sending an unwanted reminder, allowing an invalid transition, or excluding an eligible transaction. Choose plausible mistakes based on that rule. There is little value in manually changing arbitrary lines merely to accumulate demonstrations of failure.
A surviving mutation needs investigation. It may expose a missing assertion, or it may leave externally observable behavior unchanged for the component’s valid inputs. Explain which applies before adding another test. Likewise, a detected mutation can be redundant if several existing tests already catch it; a newly generated test still needs a reason to be maintained.
Keep a short review record beside the change: the agreed rule, relevant input and expected result, test command, deliberate change, observed failure reason, and clean rerun outcome. Label any unexecuted check as proposed. The record should let a reviewer assess the evidence without trusting an assistant’s summary of its own output.
This review comes after defining the task. Our guide to coding-agent bug reports and acceptance tests addresses that earlier preparation. For generated test contributions, the next step is narrower: select one consequential rule and establish whether the proposed assertion can distinguish its intended behavior from a plausible mistake. Expand the method where that evidence justifies the review and maintenance cost.