Source: [AI pilot recruitment: who do your results represent? — Skills for Business](<https://skillsforbusiness.ai/blog/ai-pilot-recruitment-results-represent/>)

[← All articles](https://skillsforbusiness.ai/blog/)

HR & people · Guide

# AI pilot recruitment: who do your results represent?

Before expanding a workplace AI pilot, record who was eligible, invited and included. Use a coverage review to keep rollout claims within the evidence.

[Skills for Business](https://skillsforbusiness.ai/blog/editorial/)1 November 20267 min read

On this page

1.  [Recruitment and random assignment answer different questions](https://skillsforbusiness.ai/blog/ai-pilot-recruitment-results-represent/#recruitment-and-random-assignment-answer-different-questions)
2.  [Selection can hide enthusiastic users too](https://skillsforbusiness.ai/blog/ai-pilot-recruitment-results-represent/#selection-can-hide-enthusiastic-users-too)
3.  [Keep the participant flow visible](https://skillsforbusiness.ai/blog/ai-pilot-recruitment-results-represent/#keep-the-participant-flow-visible)
4.  [Recruit for working conditions, not a flattering demonstration](https://skillsforbusiness.ai/blog/ai-pilot-recruitment-results-represent/#recruit-for-working-conditions-not-a-flattering-demonstration)
5.  [Keep the support arrangement attached to the result](https://skillsforbusiness.ai/blog/ai-pilot-recruitment-results-represent/#keep-the-support-arrangement-attached-to-the-result)
6.  [Match the next decision to the uncovered question](https://skillsforbusiness.ai/blog/ai-pilot-recruitment-results-represent/#match-the-next-decision-to-the-uncovered-question)
7.  [Sources](https://skillsforbusiness.ai/blog/ai-pilot-recruitment-results-represent/#sources)

What to take away

-   Random assignment within a pilot does not make its volunteers a random sample of the workforce.
-   Record participation from eligibility through analysis, including missing outcomes and differences in support.
-   Use uncovered working conditions to choose the next pilot, rather than declaring a workforce-wide result.

Before expanding a workplace AI pilot, state whose work its results describe. A successful trial with volunteers can justify a next step for similar users and conditions. It does not automatically establish the benefit for everyone who might receive the tool.

The practical remedy is to keep a participant-flow record beside the outcome report, then check which working conditions remain uncovered. This guide proposes both records for HR, learning and operational teams. They are planning aids, not a statistical certification of representativeness.

Our guide to [measuring completed work rather than AI logins](https://skillsforbusiness.ai/blog/workplace-ai-adoption-measure-completed-work/) addresses the outcome: accepted work, review and rework. Here the question comes earlier: how did people enter the pilot, and which population can the eventual result reasonably inform?

## Recruitment and random assignment answer different questions

Recruitment determines who enters a study. Random assignment determines which condition those participants receive, such as access to an AI assistant or a comparison workflow. Randomizing conditions among volunteers does not undo the process that selected those volunteers.

Elizabeth Stuart, Catherine Bradshaw and Philip Leaf explain this distinction in their [2015 Prevention Science paper on generalizing randomized trials](https://stacks.cdc.gov/view/cdc/30648/cdc_30648_DS1.pdf). Their methods and school-program case study concern whether a trial’s effects apply to a specified target population. They do not evaluate workplace AI.

A useful nuance from the paper is that a difference between participants and the target population does not by itself establish a different treatment effect. The concern is especially relevant when characteristics associated with participation also relate to how much the intervention helps. Its weighting methods depend on assumptions and observed information; a coverage table cannot reproduce that analysis.

For a business pilot, therefore, ask about plausible differences in how the workflow operates. A volunteer may already know how to check an AI-generated answer. Another colleague may need an accessible interface, different language support or permission to use the source material. These are questions to investigate, not reasons to presume that one group will perform better.

## Selection can hide enthusiastic users too

Do not assume selection always makes AI look more useful. In its [24 February 2026 experiment update](https://metr.org/blog/2026-02-24-uplift-update/), METR reported difficulty recruiting and retaining developers who did not want to work without AI. Developers also selected which tasks to submit for randomization. METR judged the resulting productivity signal unreliable and believed these selection effects likely understated speedups.

That is the researchers’ interpretation of their study, not an independent estimate of today’s tools or all developers. They also identified lower participant pay and difficulties measuring concurrent agent work. There is no justified correction factor to import into an employer’s pilot.

The lesson for recruitment is narrower: participation can depend on the comparison being requested. A frequent user may decline a no-AI condition; an inexperienced user may decline a trial that requires unpaid preparation. Record the offered conditions before interpreting the people who accepted them. Do not infer anyone’s motivation merely from nonparticipation.

## Keep the participant flow visible

The following is a **hypothetical planning example**, not an observed pilot. An operations team wants to evaluate an assistant for drafting internal service replies. It identifies 120 eligible staff, invites 80 and receives 24 acceptances. Twenty people begin; 16 provide complete records under the planned measurement rules.

| Stage | Fictional count | Question for the pilot owner |
| --- | --- | --- |
| Eligible under the stated scope | 120 | Does eligibility match the workforce for the proposed rollout? |
| Invited | 80 | Why were the other 40 not invited? |
| Accepted participation | 24 | Were working time, access and support available to those invited? |
| Began the workflow | 20 | What prevented the four other participants from starting? |
| Supplied complete outcome records | 16 | Which records are missing, and how will the analysis handle them? |

The last row is an analysis boundary, not a license to discard inconvenient attempts. Preserve incomplete and abandoned work in the operational record. Separate a missing measurement from a failed task. If the pilot randomizes conditions, report flow and missing outcomes for each condition from assignment onward; seek suitable analytical help before excluding participants or treating missing outcomes as successes or failures.

Use consistent definitions. “Started” might mean attempting the first eligible task, rather than opening an invitation. “Complete record” might require an outcome decision plus human review time. Agree those meanings before seeing results, and document later changes.

Keep reasons voluntary and proportionate. An aggregate access problem can guide a fix without distributing a named list of employees who declined. Provide a private feedback route, and avoid using this pilot record as an individual performance ranking. Unknown reasons should remain unknown.

## Recruit for working conditions, not a flattering demonstration

Nielsen Norman Group’s [November 2021 practitioner guide to recruiting research participants](https://www.nngroup.com/articles/recruiting-screening-research-candidates/) recommends defining eligibility and understanding the biases of each recruitment channel. It notes that an internal panel of experienced users can be useful while missing new-user perspectives, and that narrowly defined screening criteria can exclude suitable people.

This is user-research guidance, not evidence of an AI productivity effect. Its practical relevance is to examine the invitation route: a call in an AI enthusiasts’ channel answers a different recruitment need from invitations across the intended workflow.

For the fictional service-reply pilot, the team could prepare this coverage review:

| Intended working condition | What the current pilot covers | Next evidence to seek |
| --- | --- | --- |
| People new to the approved assistant | Not yet established | Check their access, onboarding and first-task experience. |
| Established users with immediate expert help | Covered by the initial volunteer group | Describe that help as part of the tested arrangement. |
| Staff working outside expert-support hours | Not yet covered | Offer a supported session during that shift before wider use. |
| Different approved input languages or access needs | Scope still to be confirmed | Establish requirements with actual users and check the workflow. |

These are illustrative conditions to adapt, not required quotas or a prediction of subgroup outcomes. Choose the few distinctions that could affect this particular task. Do not collect sensitive employee attributes simply to make a dashboard look comprehensive.

A box marked “covered” means there is relevant participation to inspect. It does not mean enough observations exist to estimate an effect for that group. Several participants may also share the same manager, trainer or source documents, so their experiences should not be casually described as independent replications.

## Keep the support arrangement attached to the result

Suppose the fictional volunteers have protected practice time and a specialist sitting beside them. The next rollout would offer a recording and an email address. Those are different delivery arrangements, even if the assistant and task stay the same.

Our recommendation is to record the assistance actually supplied: preparation time, examples, coaching, escalation and reviewer availability. Separate support intended for permanent use from temporary help used to discover problems. Then decide whether to retain that help or evaluate the proposed lighter arrangement.

This avoids turning an unusually well-supported pilot into an unsupported promise. It also preserves useful findings: a workflow may be worthwhile with coaching, even when the organization has not established that it works with self-service onboarding.

For a small business, the record may be a single page covering everyone who performs the task. An enterprise may need separate records for sites or service lines. Neither organization needs to claim a workforce-wide average to learn where onboarding fails or whether a bounded workflow deserves another trial.

## Match the next decision to the uncovered question

At the review meeting, pair the outcome with a scope statement. For the fictional example: “These findings concern participating service staff using this assistant, these documents and immediate specialist support. New-user and out-of-hours conditions remain untested.” Add the actual recruitment and outcome counts when available.

Then choose the smallest next step that addresses a consequential gap. If the purpose was exploratory usability research, observed obstacles may be enough to revise onboarding. If the decision is expansion to a shift with different support, recruit willing users from that shift and evaluate its intended arrangement. If the decision requires a numerical workforce benefit estimate, commission an appropriate sampling, comparison and analysis design rather than multiplying a volunteer average by total headcount.

More volunteers from the same invitation channel may improve precision within that channel without answering the missing question. Equally, one different participant cannot establish that an entire group is now represented. Document what changed in the evidence and what is still uncertain.

Before the next invitation goes out, have the operational owner and learning lead agree the eligible population, participation record and coverage gaps. Give the eventual result an explicit boundary from the start.

## Sources & further reading

Sources checked for this article. Research findings and vendor claims are distinguished in the text.

1.  [Assessing the Generalizability of Randomized Trial Results to Target Populations](https://stacks.cdc.gov/view/cdc/30648/cdc_30648_DS1.pdf)Stuart, Bradshaw and Leaf; Prevention Science / CDC Stacks · Accessed 1 November 2026
2.  [We are Changing our Developer Productivity Experiment Design](https://metr.org/blog/2026-02-24-uplift-update/)METR · Accessed 1 November 2026
3.  [Recruiting and Screening Candidates for User Research Projects](https://www.nngroup.com/articles/recruiting-screening-research-candidates/)Nielsen Norman Group · Accessed 1 November 2026

Produced with AI-assisted research and writing. Examples and recommendations are our editorial synthesis unless attributed. This article does not claim independent product testing or human expert review. [Read our editorial policy.](https://skillsforbusiness.ai/blog/editorial/)

## Keep reading

[AI-drafted interview guides: rehearse the rating criteria first](https://skillsforbusiness.ai/blog/ai-interview-guides-scoring-rehearsal/)[AI skills inventories: separate inference from demonstrated ability](https://skillsforbusiness.ai/blog/ai-skills-inventories-inference-demonstration/)