When an AI coding agent reports that tests passed after several retries, inspect the failed attempts before accepting its fix. The final green result tells you that one execution passed. It does not, by itself, explain an earlier failure or show that the patch removed its cause.
This matters because coding assistants can change code, rerun commands, adjust test setup and summarize the result within one session. A short completion message can collapse those different events into “fixed and tested.” A reviewer needs to know which revision was tested, what each attempt actually exercised and what remains unresolved.
Our editorial recommendation is to separate recovery from repair. A rerun can restore feedback after a temporary interruption. Accepting a code change requires evidence that the intended behavior is correct, with the earlier failure accounted for rather than hidden by the latest pass.
Immediate retries do not reveal every intermittent failure
A flaky test produces inconsistent outcomes under conditions developers intend to keep constant. The accessible abstract of a 2026 study by Fabian Leinen and colleagues reports an analysis of four industry-scale projects, including fresh-environment validation of failures. It finds that immediate in-job retries leave some flaky failures undetected.
The authors’ replication package makes the distinction concrete: it records failures that survived immediate reruns but subsequently passed during validation, as well as failures that were not immediately retried. It retains execution context and retry counts rather than only a final verdict.
This is CI research, not an evaluation of AI-generated patches. The full journal text was not available in the inspected sources; we use the authors’ abstract and replication documentation without importing prevalence estimates or claiming to have reproduced the study. Its relevance is methodological: a retry policy observes only part of the failure behavior.
Resource changes can alter the comparison
In its February 5, 2026 engineering report, Anthropic describes coding-agent benchmark experiments that varied resource configurations while holding the model, harness and tasks constant. More headroom reduced some infrastructure failures, but generous resources could also enable approaches that tighter limits would not support.
This is the model vendor’s own experiment, not independent validation of a workplace coding assistant. Its benchmark-specific recommendations are not universal settings for your CI workers. The practical implication we draw is to record a resource or timeout change as a changed condition, not silently credit it to a code repair.
If an agent’s second run passes after doubling memory, the team may have learned something useful about the workload. It has not isolated the effect of the source patch. Whether the application must operate within the original limit remains a requirement decision.
Read an attempt log before the success summary
Consider a hypothetical agent working on a draft-saving feature. The acceptance check expects the page to confirm that the draft was saved after the save operation completes. The following log is invented; it is not a product test or a measured agent outcome.
| Attempt on the same candidate revision | Observed outcome | What the reviewer can conclude |
|---|---|---|
| First | Browser executable unavailable; test never starts | The required browser check was not executed |
| Second | Test runs; expected saved confirmation is absent | A behavioral check failed under these conditions |
| Third | Same command passes without code edits | The result varies; the cause is not yet established |
| Fourth | Same command passes again without edits | Another passing observation, not an explanation of the failure |
Do not turn this into a success percentage. The first attempt did not execute the test, and the observations were not a designed reliability experiment. Keep the failed assertion and its trace available even if the CI interface ultimately displays green.
Two explanations might fit the second attempt. The test may inspect the page before a valid asynchronous update has completed. Alternatively, the application may mishandle a delayed save response. A rerun cannot decide between them.
The repair differs. One case calls for a test that waits for the intended completion condition. The other may require changing the application so that confirmation and persistence follow the correct sequence. Automatically adding a long sleep risks concealing either problem.
Freeze the candidate while investigating variation
Once a failure becomes intermittent, stop mixing speculative code edits with repeated executions. Record the candidate revision and the relevant uncommitted diff, then preserve the command, test selection, runtime versions, worker resources and failure details. Include whether retries are already built into the runner.
Rerun with a stated diagnostic purpose and a bounded budget appropriate to the issue. Testing a clean worker can help investigate persistent state; running the relevant group in its normal order can help investigate order dependence. State what changed between those attempts. Do not claim all conditions were identical merely because the Git commit stayed the same.
Repeated passes cannot prove that an intermittent problem is absent. The aim is to gather evidence about a plausible mechanism, not to keep trying until a green result appears. If the mechanism remains unknown when the diagnostic budget ends, report it as unresolved.
Where useful, compare the candidate with the parent revision under matched conditions. A failure on both can indicate a pre-existing problem, but does not establish that the candidate introduces no additional defect. Compare the actual symptoms and paths exercised, rather than treating all red runs as equivalent.
Repair the cause without weakening the check
CircleCI’s practitioner guide to flaky failures, last updated in December 2024, discusses asynchronous waits, timeouts, concurrency, time of day and test-order dependence. It also notes that an unreliable test can expose application behavior, not just a faulty test. This is vendor engineering guidance, not a current prevalence study or an endorsement of a specific retry feature.
For the fictional save feature, investigate the event sequence. Does the test wait for an observable completion condition? Can an earlier request overwrite a later draft? Is test data shared between cases? Which of these possibilities is supported by the trace?
A useful repair check could control when a simulated save response completes and assert the required UI state before and afterward. That makes the relevant timing condition deliberate. It should retain the application’s actual requirement, rather than simply accepting any state that makes the test pass.
Changes to assertions, skips, mocks and timeouts need review alongside application edits. Some are legitimate repairs. They also change what the evidence means. Our guide to checking that AI-generated tests catch the intended regression explains how to challenge a test’s behavioral assertion instead of relying on coverage alone.
Preserve the distinction in the review record
Ask the agent’s completion report to identify the final revision, changed files, commands and outcomes, including earlier failures relevant to the accepted change. It should distinguish an environment repair from a test repair or application repair, and name checks that did not execute.
For the hypothetical log, “browser tests passed” would omit material evidence. A more useful report would say that setup initially blocked execution, a later assertion failed intermittently on the candidate, and subsequent passes did not yet explain that failure. Once a cause is reproduced and repaired, the report can describe that mechanism and the final verification performed.
Do not allow a retry to silently replace a required check with a narrower selection. A focused test is useful during diagnosis; the repository’s required release checks still apply to the integrated revision. Any authorized exception should remain visible with its owner and scope.
The existing guide to coding-agent bug reports and acceptance tests covers defining the task before work begins. Rerun review adds a later responsibility: retain the evidence produced while the agent tries to complete it.
Start with one intermittent failure
For a small team, a short attempt record and links to traces may be enough. A larger platform team can capture the revision, runner configuration and retry history automatically, while protecting secrets and limiting access to logs. Neither approach requires storing every line in the pull-request description.
Select one recurring test failure and ask three questions before another rerun: what exactly failed, what will this attempt change, and what decision will its result inform? If the only answer is “perhaps it will turn green,” keep the failure visible and investigate the cause. A recovered pipeline is useful; an explained repair gives the team a stronger basis for accepting the AI-assisted change.