Source: [Audit AGENTS.md for the behavior each instruction changes — Skills for Business]() [← All articles](https://skillsforbusiness.ai/blog/) Development · Analysis # Audit AGENTS.md for the behavior each instruction changes Research on repository instructions shows mixed results. Audit AGENTS.md by the mistake each rule prevents, its scope, and the evidence that it remains useful. [Skills for Business](https://skillsforbusiness.ai/blog/editorial/)2 November 20267 min read On this page 1. [Read the research as different measurements](https://skillsforbusiness.ai/blog/agents-md-instructions-behavior-audit/#read-the-research-as-different-measurements) 2. [Give each rule a reason to exist](https://skillsforbusiness.ai/blog/agents-md-instructions-behavior-audit/#give-each-rule-a-reason-to-exist) 3. [Work through a fictional billing repository](https://skillsforbusiness.ai/blog/agents-md-instructions-behavior-audit/#work-through-a-fictional-billing-repository) 4. [Scope instructions without weakening requirements](https://skillsforbusiness.ai/blog/agents-md-instructions-behavior-audit/#scope-instructions-without-weakening-requirements) 5. [Inspect compliance and outcomes separately](https://skillsforbusiness.ai/blog/agents-md-instructions-behavior-audit/#inspect-compliance-and-outcomes-separately) 6. [Maintain rules when the workflow changes](https://skillsforbusiness.ai/blog/agents-md-instructions-behavior-audit/#maintain-rules-when-the-workflow-changes) 7. [Sources](https://skillsforbusiness.ai/blog/agents-md-instructions-behavior-audit/#sources) What to take away - Separate an agent following instructions from an agent delivering a correct, useful change. - Give each repository rule a specific purpose, applicable scope and observable check. - Prune redundant guidance without removing required safeguards or assuming shorter files guarantee better results. Review a repository instruction file by asking what each instruction causes the coding agent to do differently, and why that difference matters. Keep verified project constraints and useful workflow guidance. Challenge rules that merely describe an obvious directory, request unbounded exploration or repeat a tool’s existing checks. For an engineering lead, the decision is how to maintain useful instructions as the repository evolves. A complete-looking `AGENTS.md` is easy to generate; its operational value needs evidence. An agent can obey every line and still deliver the wrong change, or spend longer reaching the same result. This article proposes a behavior audit for repository instructions. The billing-service example is fictional, and we have not tested its instructions or independently benchmarked coding agents. This is a different task from documenting code intent, covered in our [self-documentation guide](https://skillsforbusiness.ai/blog/self-documenting-ai-code-best-practices/). ## Read the research as different measurements The [September 2026 revision of Gloaguen and colleagues’ study](https://arxiv.org/html/2602.11988v3) compares four agent/model configurations on SWE-bench Lite and CTXbench, including 138 tasks from 12 repositories with developer-provided context files. Generated files increased average inference cost by 20% and 23% on the two benchmarks, without a statistically significant resolution-rate improvement. Developer-provided files outperformed generated ones, but their improvement over no file was not statistically significant. The traces showed that agents generally followed instructions and performed more testing and exploration. These are Python benchmark results using Sonnet 4.5, GPT-5.2, GPT-5.1 mini and Qwen3-30B-Coder configurations. They do not establish that repository instructions are useless, or predict the cost of a current team’s assistant. The distinction between instruction compliance and successful task completion is the useful warning. A [different study by Lulla and colleagues, revised in March 2026](https://arxiv.org/html/2601.20404v2), reports lower median runtime and fewer output tokens with `AGENTS.md` across 124 pull-request tasks from 10 repositories using GPT-5.2-Codex. However, comprehensive correctness evaluation was outside its scope; a sampled manual sanity check was more limited. Faster completion and improved patch correctness are different claims. Those efficiency results cannot cancel or confirm the other paper’s resolution results. A [July 2026 follow-up by Prakhar Khatri](https://arxiv.org/html/2607.27250v1) compared no context, always-present instructions and selective retrieval, finding no detectable correctness effect in its small task set. It used 17 tasks for Codex and 15 for Claude Code across three Python repositories. Its selective condition also changed the available corpus in two repositories, and delivery channels differed between agents. It is further evidence against assuming an automatic benefit, rather than a universal verdict on instruction files. ## Give each rule a reason to exist The proposed audit starts with a sentence: **When this situation occurs, this instruction should change this action, protecting this requirement.** If the sentence cannot be completed, investigate before adding more prose. A rule may have a sound purpose without a measured productivity benefit. For example, protecting immutable release artifacts is a project obligation. It should not disappear because a small benchmark fails to show faster coding. Keep required boundaries while evaluating whether their wording and placement help the agent observe them. For optional guidance, ask what information is missing from the repository itself. “The tests are in the tests directory” offers little when the directory is plainly named. “The billing integration tests require the local ledger fixture; follow the setup in the billing test guide” supplies a potentially consequential prerequisite. Verify the prerequisite rather than promoting an assistant’s plausible guess into policy. Record a source for the rule: a maintained runbook, an approved interface contract, an actual recurring mistake or a deliberate owner decision. A generated overview is a candidate description, not evidence that every statement deserves a standing instruction. ## Work through a fictional billing repository Imagine a team maintaining a billing service with generated client libraries, local unit tests and an optional external integration environment. A proposed instruction file contains several apparently sensible rules. The following table is an editorial audit example, not an experiment or a recommended universal configuration. | Candidate instruction | Behavior to examine | Proposed disposition | | --- | --- | --- | | Read every file before editing | Broad reading even for a narrow change | Replace with relevant code, callers and tests; expand when dependencies justify it | | Edit generated client files to keep them current | Direct edits that regeneration could overwrite | Point to the schema and maintained generation workflow | | Always run every available check | External setup and unrelated checks for every task | State required gates and the changes that trigger additional checks | | Use the standard formatter | Formatting decisions already covered by configuration | Name the authoritative command; avoid copying all its style rules | | Never modify retained release artifacts | Changes to protected historical outputs | Retain the boundary and identify the protected paths precisely | Suppose the immediate task changes a calculation inside the billing service. The reviewer should be able to explain why the instruction leads the agent to the relevant caller and boundary tests. If it instead sends the agent through every generated client directory, the file may be imposing work without supplying missing knowledge. Conversely, a schema change may genuinely require regeneration and integration checks. The audit should preserve that conditional requirement. Cutting the file down to a slogan such as “run relevant tests” would leave the agent to infer a project decision it may not know. ## Scope instructions without weakening requirements Separate three questions: what is mandatory for every change, what applies to a particular component, and what applies only to the current task. Put the trigger beside a conditional instruction so the agent can determine when it applies. For the fictional repository, an owner-approved rule could say: “For billing API schema changes, follow the client-generation runbook and run its listed compatibility checks. Edit the schema source; do not hand-edit generated clients.” That preserves a specific obligation without requiring the generation workflow for a spelling correction elsewhere. [HumanLayer’s November 2025 practitioner guide](https://www.hlyr.dev/blog/writing-a-good-claude-md) recommends concise, broadly applicable root instructions, pointers to authoritative material and separate task-specific guidance. It also favors deterministic formatting tools over prose style enforcement. These are practitioner recommendations, not controlled evidence of an optimal file length. The guide also advocates a repository overview; whether a particular overview saves useful work remains a question to check. Do not adopt an arbitrary line limit as the success criterion. A necessary exception can be worth several paragraphs, while a short ambiguous prohibition can cause repeated errors. Also verify the actual assistant’s instruction-loading behavior before relying on a nested file or linked document. A link existing in Markdown does not establish that it was loaded for a particular run. ## Inspect compliance and outcomes separately For a consequential rule, identify an observable action and the outcome it is supposed to support. In the fictional generated-client case, the action is editing the schema and using the documented generator. The outcome is a consistent schema/client change that meets the compatibility contract. Evidence of the first does not establish the second. Use traces and diffs to diagnose the gap. If the agent never read the relevant instruction, adding more detail may leave the delivery problem untouched. If it read and followed the instruction but produced the wrong calculation, the missing ingredient may be a requirement, an implementation correction or a better test. Rewriting the repository overview would be an unsupported response. When evaluating an optional instruction revision, keep the model, tools, repository state and task fixed where practical. Track correctness, required checks, review effort and elapsed work. Our [guide to comparing instruction-package revisions](https://skillsforbusiness.ai/blog/skill-updates-paired-task-comparison/) explains matched comparisons and regressions in more detail. Such comparisons belong in an isolated evaluation environment; do not remove production safeguards merely to create a cleaner baseline. ## Maintain rules when the workflow changes Give important instructions a maintenance owner or a clear review trigger. A test-command change should prompt a check of the agent guidance that names it. Removing a generated directory should remove the corresponding obsolete restriction. A temporary workaround should identify the condition that will retire it. Keep the audit record compact: instruction, intended behavior, source of authority, applicable scope and most recent verification. A small team can maintain this in a change description. A larger organization can attach it to the component’s existing ownership process. Neither needs a second architecture manual solely to justify the first. Start with one instruction that repeatedly causes confusion or expensive detours. Establish the requirement, inspect what the agent actually did, and decide whether to clarify, relocate, enforce through tooling or remove the optional rule. Preserve the evidence for that decision so future edits improve the workflow instead of accumulating another layer of plausible advice. ## Sources & further reading Sources checked for this article. Research findings and vendor claims are distinguished in the text. 1. [Evaluating AGENTS.md: Are Repository-Level Context Files Helpful for Coding Agents?](https://arxiv.org/html/2602.11988v3)Gloaguen et al.; arXiv · Accessed 2 November 2026 2. [On the Impact of AGENTS.md Files on the Efficiency of AI Coding Agents](https://arxiv.org/html/2601.20404v2)Lulla et al.; arXiv · Accessed 2 November 2026 3. [Do Context Files Help Coding Agents? A Two-Agent Ablation Study on Real Repositories](https://arxiv.org/html/2607.27250v1)Prakhar Khatri; arXiv · Accessed 2 November 2026 4. [Writing a good CLAUDE.md](https://www.hlyr.dev/blog/writing-a-good-claude-md)HumanLayer · Accessed 2 November 2026 Produced with AI-assisted research and writing. Examples and recommendations are our editorial synthesis unless attributed. This article does not claim independent product testing or human expert review. [Read our editorial policy.](https://skillsforbusiness.ai/blog/editorial/) ## Keep reading [AI release notes: confirm what shipped before drafting](https://skillsforbusiness.ai/blog/ai-release-notes-confirm-shipped-changes/)[AI code review: verify the finding before applying the fix](https://skillsforbusiness.ai/blog/ai-code-review-findings-before-fixes/)