Before reporting an improvement from an AI pilot, check whether the work being measured changed. An overall acceptance rate can rise while the rate within every task category falls. The totals may be calculated correctly and still provide a poor basis for attributing the improvement to AI.
Keep the actual aggregate, the category-level counts and the task mix together. If a fixed-workload comparison would help the decision, calculate it using the same explicitly chosen weights for both periods. Label that result as a comparison for a specified mix, rather than presenting it as the outcome the team actually delivered.
This guide develops a fictional service-reply example with arithmetic checked locally using exact fractions. It proposes a review method, not an AI productivity estimate or a statistical evaluation of a real business.
Separate the outcome from the composition of the work
Our guide to measuring completed work recommends tracking accepted outcomes, review and rework across comparable tasks. Here the narrower question is what happens when the proportions of those tasks change between the baseline and the pilot.
Suppose a team drafts internal service replies. Some requests follow a standard route; others arrive with an exception already recorded at intake. The exception category might require additional source checking or an escalation. These are invented categories for this example, not a finding that all exception work is harder.
Define the categories before examining results, using information recorded before the assistance being evaluated. Keep the acceptance criteria and observation window consistent. For this example, an accepted case means the reply meets the agreed quality bar within the review window. All eligible cases remain in the denominator, including attempts that fail or are abandoned.
That definition matters. If the team quietly changes the quality threshold, drops unfinished cases or gives the later period more time to mature, a task-mix calculation will not repair the comparison. First establish that the counts describe the intended outcome.
A fictional pilot with an apparently strong aggregate gain
The team observes 100 cases before the pilot and 100 during an AI-assisted period. No random assignment is assumed. In the second period, the service happens to receive far more standard-route requests and fewer exception requests.
| Intake category | Before: accepted / eligible | Before rate | AI-assisted period: accepted / eligible | Later rate |
|---|---|---|---|---|
| Standard route | 9 / 10 | 90% | 72 / 90 | 80% |
| Exception recorded | 36 / 90 | 40% | 3 / 10 | 30% |
| All cases | 45 / 100 | 45% | 75 / 100 | 75% |
Every number in the table is fictional. We checked the subgroup and aggregate arithmetic with Python’s exact-fraction arithmetic. Both category rates decline by ten percentage points, while the overall rate rises by thirty percentage points.
The change is possible because the weights move. Before the pilot, standard-route cases account for one tenth of the work. In the later period, they account for nine tenths. The higher-rate category now contributes most of the total.
The aggregate still answers a useful operational question: what proportion of the cases received in that period reached the stated finish line? It records 75 accepted cases rather than 45. It does not establish that the AI-assisted workflow would perform better on the earlier workload, or that AI caused either the aggregate improvement or the category-level declines.
Calling this a data error would miss the problem. The issue is the interpretation attached to a valid calculation. A headline such as “AI raised acceptance from 45% to 75%” adds an attribution that the before-and-after table cannot support.
Use a common mix to ask a different descriptive question
For illustration, suppose the team wants to compare both periods on a workload that is half standard-route cases and half exceptions. This equal mix is an invented comparison target, not a recommendation that every business should use equal weights.
The before-period rate for that mix is:
0.5 × 90% + 0.5 × 40% = 65%.
The later-period rate for the same mix is:
0.5 × 80% + 0.5 × 30% = 55%.
The fixed-mix comparison falls by ten percentage points. It makes the composition issue visible without replacing the actual observed rates of 45% and 75%.
Choose a comparison mix because it matches a decision, such as an agreed reference workload for the next stage of the pilot. Document its origin and keep it fixed across the comparison. Choosing weights after seeing which ones make the pilot look attractive creates another opportunity to mislead.
A fixed-mix summary is still only descriptive here. It cannot account for unrecorded differences within the categories, staffing changes, learning, seasonal demand or other changes between periods. If a category has no observations in one period, its rate is unknown; do not fill that gap with zero or an assistant-generated estimate and present the result as measured.
The statistical lesson is older than AI
E. H. Simpson’s 1951 paper on interaction in contingency tables examines how associations in grouped data can differ from those in a combined table. Its contrasting interpretations of the same numerical structure are a reminder that context matters when deciding which relationship to report.
We inspected the original paper’s examples and discussion through a university-hosted scan. It is a statistical reference, not research on AI adoption. The workplace table above is our own hypothetical construction. Its arithmetic illustrates an aggregation reversal commonly called Simpson’s paradox; it does not supply a rule that subgroup results are always the correct answer.
The decision determines the comparison. A service owner may need the actual overall outcome to manage today’s queue, a fixed-mix summary to compare operational periods, and a properly designed evaluation to estimate the effect of introducing assistance. Preserve those questions instead of forcing one number to answer all three.
Splitting the data can also create a misleading story
Do not respond to an inconvenient aggregate by searching through subgroup definitions until a favorable result appears. In A Dirty Dozen, their 2017 KDD paper, Microsoft experimentation researchers describe segment interpretations distorted when the treatment itself changes who enters a segment. They also discuss the multiple-testing problem created by repeatedly looking for a favorable subgroup.
This is a practitioner research account of online controlled experiments, not an evaluation of workplace AI. Its relevant lesson is to inspect how a segment is formed. “Needed human escalation after using AI” is not interchangeable with an exception category recorded before assignment: assistance may itself change escalation behavior.
For our fictional comparison, retain the intake category as originally recorded. Analyze any later routing change as a separate outcome. If the assistant changes how cases are classified at intake, that classification also needs review; calling it an input does not make it unaffected by the intervention.
Optimizely’s practitioner guidance on Simpson’s paradox gives another specific example: changing the distribution between experiment variants over time can make combined results misleading. That is guidance for its experimentation setting, not a prescription for adjusting an employer’s pilot after the results arrive.
A planned randomized comparison requires its own analysis and allocation rules. The descriptive exercise here should not override them. If the question is causal, preserve the assigned population and use an appropriate design rather than treating a hand-picked weighted average as a substitute.
Give the review meeting three views and one bounded claim
Our proposed reporting package contains the actual total, category counts and rates, and any agreed fixed-mix summary. Keep the numerator, denominator, time window, acceptance rule and category definition attached to each view. Record missing classifications explicitly instead of silently removing those cases.
For the fictional pilot, a defensible descriptive statement would be: “The overall acceptance rate rose from 45% to 75% as the standard-route share rose from 10% to 90%. Acceptance declined within both recorded categories. At an illustrative equal mix, the rate would be 65% before and 55% later. These observations do not identify an AI effect.”
The next step is to investigate the change in demand and the within-category declines, then decide what comparison is needed before expansion. Small category counts also make observed rates uncertain. This arithmetic demonstration includes no confidence intervals or claim of statistical significance.
For a small business, a two-category count table may reveal an important reporting problem. An enterprise may need a more formal analysis across sites, time periods and operating conditions. In either case, avoid collecting unnecessary employee attributes or turning the exercise into individual performance ranking.
Keep this task-mix record alongside the pilot recruitment boundary. Who participated and what work they received are separate parts of the evidence. Before the next rollout decision, make both visible and match the claim to the comparison actually performed.