When a reusable agent skill produces disappointing results, first establish whether its instructions were actually loaded. The problem may be an unavailable package, a request that failed to select it, or poor execution after activation. Each needs a different remedy. Rewriting the workflow before identifying the failure can leave the original problem untouched.
Here, a skill means an instruction package built around a SKILL.md file, not a person’s workplace capability. Our proposed diagnostic method checks availability, activation and task results separately, using the same business task wherever possible.
This guide describes checks a team can run in its chosen assistant. It does not report tests of a skill or claim that every product exposes the same installation, invocation or logging controls.
The description and the body do different work
The Agent Skills specification describes progressive loading: the name and description help an agent identify relevant skills, the instruction body loads on activation, and supporting resources load as needed. It also says a description should explain what the skill does and when to use it.
That creates an important diagnostic distinction. The detailed procedure may be excellent while the description does not identify the requests it serves. Conversely, a description may attract the right task while the body fails to guide the work adequately.
Our introduction to SKILL.md work instructions explains how to write a repeatable procedure. Before revising that procedure, inspect the path that makes it available to the assistant. A readable file in a shared folder is not, by itself, evidence that the intended product has discovered or activated it as a native skill.
Establish availability in the actual product
Record the assistant product, surface, model, skill revision and relevant configuration. Check that the intended package is present through the product’s supported discovery mechanism. If the environment exposes versions or duplicate names, establish which copy would be used.
As one product-specific example, Claude Code’s documentation describes checking available skills and using direct invocation for skills that permit it. It also documents invocation controls: disable-model-invocation: true prevents automatic loading, while user-invocable: false prevents direct user invocation. These are Claude Code controls, not universal fields that every assistant implements.
A skill configured for manual use should not be judged defective simply because an ordinary request leaves it inactive. The expected behavior needs to match the configuration and the team’s intended workflow.
If the skill is missing, investigate discovery, location and configuration before editing its prose. If it is available but inactive for an eligible request, investigate selection. If it activates and then fails, examine the instructions, inputs, resources and tools required for the task.
Keep activation evidence separate from the answer
Use the strongest evidence the product makes available: a skill-invocation event, a supported trace or another documented indication that the intended content entered the run. Record the revision when that information is exposed.
An assistant saying “I used the skill” is weaker evidence than an observable loading event. A result that happens to resemble the template is also inconclusive: the model might have produced a similar answer without the package.
Where the surface does not expose activation, mark it as unobserved. You can still assess the output and test a supported explicit-invocation route, but avoid turning that into a claim that automatic selection was verified.
Use fresh sessions for comparable tests unless continuation behavior is what you intend to study. A skill already loaded earlier in a conversation can obscure whether a new request would select it from a clean start. Keep the same input files and relevant environment conditions across the comparison.
Research supports testing selection as its own step
Liu and colleagues’ April 2026 preprint, How Well Do Agentic Skills Work in the Wild, varies whether skills are explicitly loaded, selected from available candidates or retrieved from a larger collection. Its experiments distinguish failure to load useful skills from difficulty using their content, and show that the harness and evaluation setting matter.
Those are findings from the evaluated models, tasks and harnesses, not failure-rate estimates for your assistant or business workflow. We have not independently replicated the study. The useful design lesson is to preserve the distinction between finding a skill and benefiting from it when constructing a local evaluation.
Anthropic’s March 2026 skill-creator practitioner article similarly discusses output evaluation and description tuning as separate concerns. It describes overly broad descriptions causing unwanted activation and overly narrow ones missing relevant requests. That is vendor guidance and an account of its tooling, not independent validation of a team’s particular skill.
A hypothetical supplier-comparison skill
Imagine a team with a skill that turns supplied vendor quotes into a comparison brief. It should preserve the quoted cost basis, identify missing information and organize questions for the buyer. It does not select a supplier or send messages. This is a fictional workflow and proposed evaluation, not an observed deployment.
The team first defines what the skill is intended to handle. Then it writes requests that test the boundary:
| Request | Expected behavior |
|---|---|
| Compare the attached quotes and list unresolved cost assumptions | Select the comparison skill and produce the brief |
| Prepare a comparison for the buyer without naming the skill | Select it from the natural business request |
| Polish this short email to a supplier | Use the ordinary writing workflow rather than the comparison skill |
| Explain what a supplier reference means | Answer the question without starting quote comparison |
| Compare the quotes when one referenced attachment is missing | Preserve the missing input rather than inventing its contents |
| Invoke the comparison skill explicitly, where supported | Load the intended procedure and assess its execution |
These cases test more than keyword matching. The word “supplier” can appear in work the skill should not handle, while a valid comparison request might use different wording. A good description communicates the job and its boundary without claiming every adjacent task.
For the positive cases, define output checks before running them. Does each cost retain its stated period and scope? Are missing figures left unresolved? Does the brief distinguish supplied facts from questions? Those checks judge the work, separately from whether activation occurred.
Use the comparison to locate the next change
Run the natural request and the supported explicit-invocation version against the same fictional inputs. Our proposed interpretation table is a way to choose the next investigation, not a claim that one run establishes a reliable diagnosis.
| Observed result | What to investigate next |
|---|---|
| The skill is unavailable | Discovery, installation scope or configuration |
| Explicit invocation works, but the natural request does not activate it | Description clarity, request coverage and competing skills |
| Both routes load the skill and produce the same defect | Instructions, source inputs, resource access or execution |
| The skill loads for unrelated requests | An overly broad selection boundary or overlap with another skill |
| The output passes without observable activation | Output quality is established for that case; activation remains unverified |
Repeat important cases to see whether the pattern persists. Preserve mixed results rather than reporting only the successful run. If another skill overlaps, test with the set employees normally have available; an isolated demonstration may conceal competition in the real environment.
A no-skill comparison can help assess added value when the environment supports a meaningful baseline. Compare the actual business criteria, review effort and required process fidelity. Do not treat activation frequency as the objective: frequent loading is unhelpful if the wrong requests trigger it or the result requires more correction.
Change one layer, then retest the boundary
If availability and invocation settings are correct but relevant requests are missed, revise the description first. Retest both the positive cases and the neighboring requests that should remain outside scope. Adding a long list of popular keywords may increase activation while making selection less precise.
If the skill loads and mishandles quoted facts, fix that execution problem and retain the activation cases. When changing referenced resources, also check that the assistant can access the intended revision. A stronger description cannot repair an unavailable input file.
Keep the test cases with the skill’s maintenance record. Record what changed, which product and model were used, the observed activation evidence and the task results. The cases in this article are proposals; running them in a real assistant would be a separate verification exercise.
For the next skill revision, choose a few realistic requests and nearby exclusions before editing. Establish where the failure occurs, change the relevant layer and rerun the comparison. That gives the team a reason for the change beyond whether the new wording sounds more authoritative.