Give a coding agent a bug it can reproduce, a way to run the relevant checks and an observable definition of the fix. Before expanding the task or changing models, establish whether the agent can distinguish the original failure from an environment problem.
For an engineering lead, this turns task preparation into a small evidence check. Can someone reproduce the unwanted behavior at a known commit? Does the proposed regression test fail because of that behavior? Can the same environment run the checks that should remain green?
This article proposes a practical intake and review method for bounded bug fixes. It is not a benchmark of coding products, a claim about productivity gains or a requirement to specify every implementation detail before an agent can start.
Why the task and its environment belong together
GitHub’s guidance for Copilot cloud agent recommends clearly scoped tasks, complete acceptance criteria and relevant directions about the codebase. It also describes repository instructions for building, testing and validating changes. These are vendor recommendations, not an independent estimate of how much better an agent will perform.
The useful implication is to make the task executable. An issue can explain the business problem clearly while still leaving the agent unable to install dependencies, obtain a safe fixture or run the relevant test.
Anthropic’s January 2026 engineering guide to agent evaluations makes a related distinction between an agent’s account of its work and the outcome in the environment. For coding, it emphasizes specified tasks, stable environments and tests. It also recommends examining execution records where they add evidence. This is practitioner guidance from an agent vendor, not a controlled trial of the intake method below.
Neither source establishes that a better issue description can compensate for an unavailable service or an unstated business rule. Those are inputs the team must resolve or explicitly leave outside the task.
Borrow a useful distinction from SWE-bench
The SWE-bench paper, first submitted in October 2023 and revised in November 2024, describes software tasks built from repository issues and associated pull requests. Its construction process runs tests before and after a solution and checks for a meaningful change from failure to success.
The evaluation also distinguishes tests that must become passing from tests that must remain passing. That provides a useful way to discuss a bug fix: demonstrate the new behavior while preserving the relevant existing behavior.
This is a borrowing from benchmark design, not evidence that an internal task will be solved at any particular rate. The original paper’s tasks were Python-based, and its historical model results should not be treated as current agent performance. The authors also acknowledge that passing execution-based tests does not establish every aspect of software quality.
For a business repository, the goal is a local acceptance record. It should explain the particular defect and what was checked, without turning a successful patch into a broad reliability claim.
Start with a small reproduction and a known baseline
Consider a hypothetical application that exports customer invoices. A user reports that the exported file puts an invoice total in the wrong column when the company name contains a comma.
“Fix CSV export” leaves several questions open. Which export route? Which input? Which consumer shows the problem? Does the error occur in the generated file or only when another application imports it?
Prepare a fictional record containing a company such as “North, South Ltd” and a known invoice amount. Identify the export operation and record what happens when its output is parsed using the project’s agreed CSV reader. State the expected company field and amount. This is an invented example; no application was tested for this article.
Keep the reproduction narrow. It should not require the agent to access real customer invoices, repair unrelated imports or redesign the reporting system. If the problem depends on a particular downstream consumer, include that compatibility requirement explicitly rather than assuming every parser behaves identically.
Record the starting commit and relevant runtime. Otherwise a reviewer may compare a result from one code state with a failure reported against another.
Establish which kind of failure you have
Run the smallest relevant existing check before changing production code. Then introduce or identify a regression check for the reported behavior. The order is useful because it separates the defect from problems in the test environment.
Our proposed intake classification is simple:
| Observation | What it establishes | Useful next action |
|---|---|---|
| The runner cannot start | The environment is not ready to assess the bug | Repair setup or report the blocker |
| Existing relevant tests pass, but the reproduction fails | There is a bounded behavior gap to investigate | Add a check for that gap |
| The new test fails before reaching the behavior | The check does not yet demonstrate the defect | Fix the fixture or test setup |
| The new test fails on the expected output | There is evidence the check detects the reported behavior | Implement and compare results |
These categories are a proposed working method, not an experimentally validated taxonomy.
A missing browser binary, an unavailable database and an incorrect exported field may all produce a red build. They justify different actions. Ask the agent to report which observation it has, including the command and the relevant failure message.
Do not ask it to remove an assertion merely because the environment cannot satisfy it. If a check cannot run, record that limitation separately from the result of checks that did run.
Define acceptance without prescribing the entire patch
A good acceptance condition describes behavior at a meaningful boundary. For the hypothetical export bug, it might require that the agreed reader recovers the exact company name and amount, with the expected number of columns.
That condition leaves room to choose a suitable implementation. Requiring a particular helper name or an arbitrary internal call sequence would constrain the solution without necessarily protecting user behavior.
Also specify nearby cases worth preserving. A company name without punctuation should continue to work. A field containing a quotation mark or a line break may matter if the existing export contract permits it. Choose cases based on that contract rather than adding every conceivable input to a small bug fix.
Keep product decisions visible. If the team has not decided which CSV dialect or downstream application it supports, the agent should not silently make that decision while fixing the example.
For a user-interface defect, the acceptance boundary might instead be a visible state and a real interaction. A unit test can help, but the evidence should match what the user actually experiences.
Put a compact task record in the issue
The following fields are our proposed issue template. Use them to make review easier, not to create paperwork for every minor edit.
| Field | What to include |
|---|---|
| Problem | Concrete trigger, current behavior and practical impact |
| Baseline | Repository, base commit and relevant runtime |
| Reproduction | Safe fixture and steps or command |
| Acceptance | Observable expected result and relevant preserved behavior |
| Verification | Focused check, nearby regression checks and any integration check |
| Boundaries | What the task may change and which decisions need an owner |
| Blockers | Known missing services, permissions or unresolved requirements |
Point to maintained repository instructions for common setup instead of copying them into every issue. A reusable agent skill can preserve the investigation procedure, while the issue supplies the current defect and evidence.
If the agent discovers that the original scope is wrong, let it propose a revised task. A narrow issue is helpful only while it accurately describes the problem.
Review the evidence alongside the diff
A pull request should make the before-and-after comparison easy to assess. Ask for the reproduction result, the focused test result, nearby regression results and any checks that could not run. Include the commands or links to their logs.
Review the test changes themselves. Does the new assertion check the intended behavior? Did an existing test become less demanding? Is the fixture realistic enough to expose the defect without importing private data?
Then review implementation concerns that the tests may not cover: maintainability, dependency changes, compatibility and behavior outside the selected examples. A passing check is evidence about what it exercises. It is not a complete argument for merging.
Separate repair work from the user-facing fix when it matters to interpretation. If setup was broken before the agent started, explain that fact. Otherwise reviewers may attribute the time spent restoring the environment to the difficulty of the bug or the capability of the model.
Improve task preparation from actual failures
After several bounded tasks, review where work became blocked. Were fixtures missing? Did instructions name an outdated command? Were acceptance criteria repeatedly clarified after implementation?
Those observations can guide changes to the repository’s setup documentation and issue template. Keep them separate from model comparisons. A workflow with different inputs or a repaired environment is a different comparison condition.
Our model-upgrade evaluation guide addresses how to compare configurations. This task-level record gives that later evaluation something more dependable to measure.
For the next bug, spend a few minutes establishing the baseline and one meaningful failure before asking for a patch. If the defect cannot yet be distinguished from setup noise, resolve that uncertainty first. The result is a task whose completion a reviewer can actually verify.