Before rolling out an agent skill update, compare it with the retained version on the same tasks. Record which tasks improve, which stop working and which remain unresolved. A higher overall pass count is useful, but it should not hide the loss of a behavior the business depends on.
This guide proposes an evaluation method for teams maintaining reusable agent instructions in SKILL.md packages. Its release-note example and result counts are invented. We have not executed either skill version, tested an assistant product or observed a customer rollout for this article.
Compare the package that actually changes behavior
The Agent Skills specification defines a skill as a directory containing SKILL.md, with optional scripts, references and assets. Comparing only the main Markdown file can miss a changed template or helper script that affects the result. Retain the complete old and candidate packages, with a version identifier or content hashes that make each recoverable.
Write a narrow change statement before editing. For a fictional release-note skill, the goal might be to make notices easier for nontechnical readers while preserving the distinction between shipped fixes and planned features. That gives the evaluation two requirements: improve accessibility of the wording and preserve delivery status.
Keep the assistant product, model, tool access, inputs and relevant settings steady during the comparison. The software around the model—often called the harness—also affects how skills are discovered and executed. If you change the model and the skill together, you can assess the combined setup, but you cannot confidently attribute the difference to the skill edit alone.
Research supports matched comparisons and measured revision
The June 2026 revision of SkillsBench compares matched tasks with and without curated skills, using containerized environments and deterministic verifiers. Its results vary with model and harness configuration. The paper also notes that terminal-based tasks and imperfect controls limit generalization. This supports the comparison principle, not a forecast of gains from a particular business skill.
SkillRevise, revised in September 2026, studies execution-grounded revision: diagnose a candidate from execution evidence, edit it, rerun it and select among evaluated versions. Its method does not automatically return the latest generated revision. The authors discuss dependence on verifiers, including unstable checks and checks that cover only part of the requirements. We do not reuse its benchmark improvements as expected results for a team rollout.
Anthropic’s March 2026 skill-creator announcement describes comparisons between skill versions or skill versus no skill, alongside timing and token metrics. This is vendor-reported functionality and guidance; we have not independently verified its operation on any assistant surface.
Our recommendation is to borrow the discipline without assuming the research infrastructure transfers unchanged: preserve a comparator, define acceptance and inspect actual outputs and execution evidence.
Prepare cases that protect existing behavior
Build a small case set from the skill’s intended work. Include the failure motivating the revision, representative successful tasks and inconvenient cases that exercise important boundaries. For the fictional release-note skill, include a shipped bug fix, a mix of released and planned changes, a request for a nontechnical audience and a source record with no confirmed release.
Define the expected behavior before running the candidate. A mixed-status notice must not describe a planned feature as available. A plain-language notice must preserve meaningful limitations. When the source contains no confirmed release, the output should keep that uncertainty visible rather than inventing a completed change.
Check functional constraints directly where possible. Review status labels against the source record, required sections against the agreed format and unsupported claims against the supplied evidence. For more subjective criteria such as readability, use an explicit rubric and compare outputs without version labels where practical. An AI preference judgment can inform review, but it does not replace a factual status check.
Read the transitions behind the average
The following invented comparison contains twelve tasks. It illustrates how a candidate can improve the aggregate while breaking previously successful work; these are hypothetical outcomes, not test results.
| Retained version | Candidate version | Invented task count | Review question |
|---|---|---|---|
| Pass | Pass | 5 | Did effort or output quality change despite both passing? |
| Fail | Pass | 3 | Are the gains real under the same acceptance criteria? |
| Pass | Fail | 2 | Which previously satisfied requirements were lost? |
| Fail | Fail | 2 | What remains outside reliable operation? |
The retained version passes seven tasks and the candidate eight. Reporting only that improvement would conceal two regressions. Suppose one regression turns a planned feature into a shipped claim. Under a proposed release rule that forbids unsupported availability statements, that candidate should not roll out merely because three other cases improved.
Do not treat every failure as equally consequential. Agree on essential constraints and acceptable tradeoffs before seeing the comparison. A formatting defect, an unsupported claim and an unavailable tool require different explanations. Keep failures visible even if a summary metric weights them differently.
Keep variability and execution failures visible
Use fresh sessions or isolated environments so one version does not inherit the other’s conversation or generated files. Preserve prompts, input versions, outputs and the execution record for each condition. Confirm that the intended package was available and, where observable, actually loaded; otherwise record activation as unobserved.
Agent runs can vary. Repeat important or unstable cases under a stated run budget rather than treating one favorable output as a stable property. Keep every attempt’s outcome. Selecting the best candidate run while retaining the first baseline run creates a misleading comparison.
When a run fails, distinguish an instruction problem from an environment problem. If a required tool is unavailable in one condition, repair or explain the mismatch before crediting the other version with a behavioral gain. Likewise, a completed artifact does not establish that every acceptance condition passed.
Record time, usage and correction effort alongside quality. A shorter output can still require more factual repair. Compare efficiency among outputs that satisfy the essential requirements, and include the cost of repeated attempts needed to get there. Do not convert a small test set into a precise organization-wide savings estimate.
Save fresh cases for the rollout decision
Examples used repeatedly to rewrite the skill become development material. Keep additional cases that the editor has not used to tune the candidate. They should test the same intended task family with different inputs, rather than introduce unrelated requirements after the fact.
Run the retained and candidate packages on those cases too. If you inspect a failure, revise against it and rerun, record that it has joined the development set. You may still use it as a regression check, but it no longer provides the same evidence about unseen work.
For a team without a large evaluation system, a few carefully chosen cases and an honest record are a useful start. They are a decision aid, not statistical proof. Expand the evidence where errors are consequential or the workflow spans substantially different task types.
Roll out a recoverable version with a stated scope
Save the candidate package, comparison results, known failures and the conditions under which it was evaluated. Identify who maintains it and what would trigger a pause or return to the retained version. Keep rollback possible without overwriting the previous package or its evaluation record.
A model change, tool change or new document format can justify another comparison even when the skill text stays unchanged. Conversely, a failed candidate does not require abandoning the whole skill: revise the narrow behavior that regressed, then rerun the cases that depend on it.
Our skill activation guide covers diagnosing whether instructions were selected and loaded. Once that boundary is understood, evaluate the revision as a behavioral change. Keep the new successes, lost successes and unresolved cases separate when deciding whether the update is ready for the stated workflow.