Before using an AI judge to choose between two assistant configurations, check whether its verdict survives reversing the order of the same two outputs. Keep the outputs fixed, exchange their display positions and map the verdict back to the original candidate identities. A judge that keeps choosing the first response has changed its preferred candidate when the order changes.
Use this as a diagnostic, alongside repeated runs and checks against an agreed reference. It can reveal an evaluation problem before a team mistakes it for an improvement in its assistant. It cannot establish that a stable winner is correct, that position bias is the only source of disagreement or that a workflow is ready for release.
This guide develops a fictional comparison log and a proposed review process. We checked the identity mapping in the invented log with a local Python exercise. We did not call a judge model, evaluate assistant outputs or measure a product’s bias.
Separate the candidate from its display label
An AI judge is a model asked to evaluate another output against criteria. In a pairwise comparison, it sees two responses and selects one, declares a tie or returns another permitted disposition. Teams may use that result to compare prompts, models or versions of an agent skill.
The display labels are temporary. If the retained configuration is shown as A in one request and B in another, the evaluation record must preserve that mapping. Otherwise, a tally of “A wins” can measure a position rather than the configuration the team intends to assess.
Suppose a service team compares replies from a retained setup and a proposed revision. Call their immutable output identities Old and New. Those identities should remain attached to the stored response text even when the judge sees neutral labels A and B. Keep generator names hidden where feasible, without losing the mapping in the evaluation record.
Our paired skill-update guide concerns comparing matched tasks and retaining regressions. This article addresses a prerequisite when a model supplies the comparison judgments: whether the measurement changes with presentation.
What the research supports
Zheng and colleagues’ NeurIPS 2023 paper, Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena, studies position and other biases in historical judges. Its proposed conservative approach compares both answer orders and declares a winner only when both prefer the same answer. The paper treats inconsistent outcomes as ties.
That research emphasizes helpfulness and combines several quality dimensions; it is not a validation of today’s judges for a company’s support policy. Our proposed log below retains order disagreement separately from an explicit tie, so a later reviewer can inspect the reason a pair produced no winner.
Shi and colleagues’ December 2025 study, Judging the Judges, distinguishes repetition stability, position consistency and preference fairness. Its benchmark analysis finds variation across judges and tasks. Its scope is limited by the models, benchmarks and comparison formats studied. These are reasons to test your actual evaluator, not to import a historical bias rate into a current business case.
Define the judgment before testing its stability
Write the rubric before collecting preferences. For the fictional service-reply workflow, suppose a response must preserve the supplied delivery status, avoid inventing a date and give the customer a usable next step. A polished reply that invents a delivery commitment fails the required factual boundary.
Specify whether the judge is deciding factual acceptability, communication quality among acceptable replies or overall preference. Those questions can produce different winners. If both replies violate a mandatory requirement, choosing the less objectionable one should not turn it into an accepted answer.
Provide the relevant source material and an explicit option for insufficient evidence where the task requires it. Preserve the rubric and reference versions with the outputs. Do not give one ordering an updated policy excerpt or a longer context window and interpret the resulting difference as an order effect.
OpenAI’s current evaluation guidance identifies response order and verbosity as challenges for model graders and recommends validating agreement with human annotations. That is vendor guidance, not a certificate that a particular judge or prompt is valid for this workflow.
A fictional log that exposes a mapping mistake
In this invented exercise, the first request displays A = Old and B = New. The reversed request displays A = New and B = Old. Each row represents a different fictional task pair, and the judge selections are made-up values rather than model observations.
| Pair | First order selection | Reversed order selection | Interpretation after identity mapping |
|---|---|---|---|
| P1 | A | B | Old preferred in both orders |
| P2 | B | A | New preferred in both orders |
| P3 | A | A | Different candidates preferred; order disagreement |
| P4 | Tie | Tie | Explicit tie in both orders |
| P5 | Invalid output | A | Incomplete comparison; no two-order verdict |
We executed a small Python check mapping each valid slot back to Old or New and asserting these five dispositions. That verifies the example’s bookkeeping only. It does not test whether a model would produce these selections or whether Old or New deserves to win.
P1 is easy to misread if the logging system compares the raw letters: A followed by B looks inconsistent, yet both select Old. P3 has the opposite trap: two A selections look repeatable, yet the preferred candidate changes.
P5 should remain incomplete. Do not convert a parser failure, timeout or missing verdict into a tie, and do not silently select the one valid result. If the workflow retries, record the attempts and apply a retry rule chosen in advance. Retrying only until a favored candidate wins would change the evaluation procedure.
Distinguish order disagreement from ordinary variability
One comparison in each order can flag an inconsistency, but it does not by itself establish a persistent positional preference. A model can vary even when given the identical request. Repeat a subset in the same order as well as in reversed order, using a fixed configuration and fresh independent requests where the interface permits them.
Record any uncertainty about independence. A hosted assistant may inherit conversation context, memory or settings that the reviewer cannot fully control. Do not describe separate chat windows as a controlled experiment without establishing the relevant conditions.
Keep the task, two candidate texts, rubric and reference material unchanged across the order check. Record the judge identifier, settings, prompt version, response, parsed verdict and candidate mapping. A compact log should allow a reviewer to reproduce the interpretation without relying on a dashboard’s winner color.
Report consistent Old wins, consistent New wins, explicit ties, order disagreements and incomplete pairs separately. The denominator for a win-rate summary should be visible. Dropping unresolved pairs can make a result look cleaner while hiding where the judge is least dependable.
Stable decisions still need a validity check
A judge can select the same unsupported reply in both positions. Order consistency is therefore an evaluation property, not proof of factual accuracy. Compare selected judgments with a reference established through an appropriate review process, including clear failures, acceptable alternatives and cases requiring uncertainty.
Eugene Yan’s August 2024 practitioner synthesis on LLM evaluators discusses choosing evaluation methods and aligning them with the criteria being measured. It also reviews position and other biases. This is a synthesis of research, not a separate replication of the studies cited here or validation of our proposed log.
For the fictional service case, inspect whether a preference explanation cites the actual delivery record. If both orders reward an invented date because it sounds helpful, the problem is substantive. Reversing the answers has not repaired the rubric or the judge’s use of evidence.
Similarly, do not rewrite a longer candidate merely to equalize its length during an order test: that changes the object being evaluated. If length sensitivity is a separate concern, design and label a separate diagnostic. Keep factual requirements independent of stylistic preference.
Decide how unresolved comparisons affect rollout
Choose the disposition rule before comparing versions. A cautious initial workflow can retain two-order agreement as one source of preference evidence and route disagreements or incomplete pairs for review. It should preserve the reason for escalation rather than compressing every non-win into a single tie count.
A small team can start with a handful of approved task pairs and inspect every judgment. A larger evaluation can automate the identity mapping and disposition checks while sampling reference agreement and investigating failures. Neither scale supplies a universal acceptable disagreement threshold; the intended decision and consequences determine what evidence is needed.
Keep the judge configuration versioned alongside the system being evaluated. If the evaluator changes between releases, a change in measured quality may reflect the instrument as well as the candidate. Recheck retained reference cases before interpreting a new trend.
Start with one existing comparison. Preserve the two output identities, reverse their display order and inspect both the raw selections and the mapped verdicts. If the interpretation is unstable, resolve the evaluation issue before using that winner to justify a production change.