Before accepting an A/B test implemented by an AI coding agent, trace how an eligible user becomes an assigned unit, a logged observation and a row in the final analysis. A conversion chart can look convincing even when one variant loses users along that path.
For an engineering lead, the immediate decision is whether the implementation produces evidence the business can interpret. A request such as “split traffic equally and report which page converts better” leaves important behavior unspecified. The assistant may build both pages correctly while giving their logging or filtering different meanings.
Our recommendation is to review the measurement path alongside the visible feature. The worked example below is fictional. Its counts are invented teaching data, and the proposed checks have not been run against a product or AI model for this article.
Define the unit before asking for a random split
Write down what receives a variant. It could be an account, a person, a browser or a session. Those units answer different questions. If a business account has several users, assigning each browser separately may expose colleagues to both variants. Whether that is acceptable depends on the decision being studied.
Also define eligibility, the intended allocation, the event that records exposure, and the outcome window. State how repeat visits, missing identifiers and identity changes are handled. The implementation needs an explicit policy for an anonymous visitor who later signs in.
Give the agent this contract before it writes the assignment helper. For example, a fictional pilot could assign each eligible account to one of two variants at equal probability, preserve that assignment for the experiment, and count each account once in the primary outcome. That is a proposed design for this example, not a universal choice.
Distinguish the assignment record from observations of what happened afterward. A user can be assigned a variant and then encounter an error before the page sends its exposure event. Keeping those stages identifiable makes that loss visible during diagnosis. Access and retention should follow the organization’s existing data rules.
Use the research as a diagnostic foundation
Fabijan and colleagues’ KDD 2019 paper developed a taxonomy of sample ratio mismatch, or SRM, using mixed methods across four companies. Its methods included internal documentation, interviews and historical experiments. The paper describes failures across assignment, execution, logging and analysis. Its industrial case evidence informs debugging, while its historical incidence figures cannot establish the failure rate of today’s AI-written software. Read the author preprint.
An SRM check asks whether observed group counts are compatible with the planned allocation under the check’s assumptions. Random assignment does not promise exactly equal counts. An ordinary small difference can arise by chance. Configure the diagnostic through the team’s established experimentation method, including its monitoring policy, rather than inventing a percentage tolerance in the generated code.
Statsig’s practitioner guide explains that an imbalance can originate in assignment or in later measurement and processing. It gives examples involving exposure timing, crashes and deleted or duplicated data. This is guidance from an experimentation vendor. We use the failure mechanisms without adopting its product assurances or treating them as independent validation. Read Statsig’s guide.
Follow one fictional cohort through three stages
Suppose an internal portal experiment starts with 1,000 eligible accounts. In this invented cohort, assignment happens to produce exactly 500 accounts per variant. Both variants have 50 accounts completing the target action.
The treatment page logs exposure only after a new panel finishes loading. A defect prevents 100 treatment accounts from sending that event. For this teaching example, all 100 are accounts that did not complete the target action. The analyst’s generated query includes only accounts with a logged exposure.
| Stage | Control accounts | Treatment accounts | What changed |
|---|---|---|---|
| Assignment record | 500 | 500 | Initial allocation |
| Exposure log | 500 | 400 | 100 treatment accounts absent |
| Final analysis | 500 | 400 | Query retains the logging loss |
| Completing accounts in analysis | 50 | 50 | Completion counts happen to match |
The final dashboard reports 50 divided by 500, or 10%, for control and 50 divided by 400, or 12.5%, for treatment. That is a difference of 2.5 percentage points and an apparent relative uplift of 25%. Against the full assigned cohort in this fictional example, both rates are 10%.
This arithmetic illustrates selection, not a statistical efficacy result. Real missing accounts may have unknown outcomes, so their conversion rate cannot simply be assumed. The team would need evidence about what was lost and an appropriate analysis before drawing conclusions.
The stage comparison narrows the investigation. Here, assignment counts agree and the first discrepancy appears in exposure logging. Rewriting the randomization helper would not address that evidence. Inspect the panel load condition and event path, then verify how the analysis query inherits the missing records.
Ask the coding agent for acceptance evidence at the boundaries
Turn the measurement contract into concrete review cases. For the fictional account-level design, revisit the same account through the supported entry points and verify that its recorded assignment remains consistent. Check that retries do not create additional units in the denominator. Use a controlled fixture with expected identities and counts.
Next, delay or fail the treatment panel deliberately in an authorized test environment. Inspect the assignment record, exposure events and analysis output. The check should reveal the failed delivery and retain enough information to reconcile it. A screenshot showing both variants provides little evidence about this path.
Inspect generated SQL as part of the implementation. A join to a later event table, an unexpected inner join, or a filter requiring a successful page render can change the analyzed population. Ask the agent to explain every exclusion and show where the eligibility rule is evaluated. The business owner should approve the intended population before anyone inspects uplift.
Our guide to SQL join fan-out covers another denominator failure. In this experiment, check both directions: duplicated event rows can inflate counts, while missing event rows can remove accounts. Preserve unit-level uniqueness through aggregation instead of relying on a plausible total.
Diagnose a warning before changing the sample
Keep the original allocation, code revision, counts and diagnostic output available when a warning appears. Compare stages and relevant segments using a stated investigation plan. Segments can locate a defect, but repeatedly slicing until a favorable subset appears creates a different analysis problem.
Lukas Vermeer’s SRM Checker FAQ emphasizes that the remedy depends on the root cause. Its diagnostic guidance includes comparing broader and filtered populations and examining differences across pipelines. We draw the operational recommendation to repair and explain the first observed divergence before accepting the result. Read the FAQ.
Avoid trimming the larger group merely to make counts equal. Equal totals would leave the original loss unexplained. If a repair changes measurement during a running experiment, document when it took effect and obtain an analysis decision about the affected period. A fresh experiment may be necessary. The appropriate remedy depends on the mechanism and the decision’s stakes.
A passing ratio check also leaves other questions open. Both groups could lose records equally, an outcome could be misdefined, or assignment could be at the wrong unit. Review the metric and implementation contract together. For the next AI-built experiment, require one trace from eligibility through analysis, with known fixture outcomes and deliberately interrupted logging. That gives a reviewer something concrete to inspect before a dashboard becomes a rollout decision.