Before putting a smaller AI model at the start of a business workflow, define what happens when its answer is insufficient. Some cases can move to a stronger model. Others need a missing attachment, a business decision or a stop. The value of routing depends on making those distinctions reliably and counting the full path to accepted work.
Anthropic’s 7 October 2026 Haiku 5.5 introduction positions the model for high-volume, cost-sensitive work, including classification and summarization. That is vendor positioning, not independent evidence that a particular business task should start there. We have not tested Haiku 5.5 or any model pair for this article.
This guide proposes an escalation policy, a fictional workflow and a cost example. The objective is a useful model-selection decision within a defined process. It is not a recommendation to add multiple models to every assistant.
Distinguish selection before a call from escalation afterward
A router can select a model before generating an answer. A cascade can start with one model, assess its answer and call another if needed. These designs have different costs: the second may pay for an initial answer that the workflow ultimately discards.
RouteLLM studies learned selection between stronger and weaker models using preference data and a decision threshold. Its comparative prediction concerns which model is likely to produce a preferred answer. That is different from establishing that an answer meets a company’s factual or operational requirements. The paper’s evaluated model pairs and benchmarks do not validate current Haiku routing or guarantee business savings.
The earlier FrugalGPT paper studies cascades that score a generated answer and query another service when the score is insufficient. It uses learned scoring and labeled examples, and notes the importance of training data resembling future queries. These are 2023 experiments, not current prices or error rates. We do not reuse their reported savings percentages.
Our practical recommendation is to name the design explicitly. “Use a small model for routine cases” is incomplete until the team specifies how cases are selected, how results are accepted and which path handles exceptions.
Start with a bounded workflow and a visible acceptance rule
Imagine a fictional operations team preparing internal briefs from customer order-change requests. The assistant extracts the order identifier, requested change and relevant dates from supplied records, then prepares a draft for an employee. It does not amend the order or promise a delivery date.
For this proposed workflow, acceptance requires the extracted fields to match the supplied records, the draft to distinguish a request from a confirmed change, and unresolved information to remain visible. The team can inspect those criteria regardless of which model produced the result.
Choose a smaller model only for the slice of work where evaluation supports that choice. “Routine” should describe an observed class of cases with defined inputs and acceptance criteria. It should not mean a short prompt or an answer that sounds confident.
Anthropic’s Building effective agents describes routing for distinct task categories when classification can be handled accurately, including with conventional algorithms. It is a practitioner account, and the page warns that its tooling discussion has changed since its original December 2024 publication. We use the architectural distinction, not its historical product examples as current implementation instructions.
Give different failures different destinations
The following policy is an editorial proposal for the fictional order brief. It is not a tested configuration or a universal division of work between small and large models.
| Condition | Proposed next step | What must remain true |
|---|---|---|
| Complete records and all acceptance checks pass | Retain the draft for the normal employee review | Preserve evidence references and unresolved states |
| Complete records, but a defined synthesis check fails | Try the approved stronger-model path | Apply the same acceptance criteria afterward |
| A referenced attachment is absent | Request the missing input | Do not infer its contents from either model |
| Two authoritative records conflict | Route the conflict to the responsible business owner | Preserve both records and the disagreement |
| The requested action exceeds the workflow’s authority | Stop or use the separate authorization process | Model escalation does not expand permissions |
The reason for escalation matters more than its label. A stronger model might help relate several supplied records, but it cannot establish a missing confirmation simply by generating more text. If the defect is an extraction-service failure, repair or rerun that service rather than treating another language-model call as the default remedy.
A valid output structure is one check, not complete acceptance. A draft can contain every required field while putting the wrong order identifier in it. Similarly, asking the first model whether it is confident does not substitute for evaluating the business criteria against examples with known outcomes.
Account for the calls that did not produce the final answer
Consider an invented batch of ten tasks. Assume the smaller-model call costs one unit per task, a stronger-model call costs five, and routing plus automated checks cost three units for the whole batch. These are planning units, not API prices or measured usage. Human review and engineering maintenance are excluded from this arithmetic and must be added in a real comparison.
If three tasks escalate after the smaller call, the batch costs 28 units: 10 + 15 + 3. Sending every task directly to the stronger model would cost 50 units: 10 × 5 under the same invented assumptions.
If eight tasks escalate, the routed batch instead costs 53 units: 10 + 40 + 3. The cheaper first call has not made that path cheaper overall. Actual prompts, output lengths, caching conditions and verification methods will change the calculation; use observed usage and the applicable price schedule when evaluating a real integration.
These examples establish arithmetic, not quality. The 28-unit path is not acceptable merely because it costs less. If it lets unsupported changes through, the team has purchased a different result. Compare the quality requirement and the total cost of meeting it together.
Also measure elapsed time along the complete path. A fast initial answer followed by checks, a second generation and another review may miss the workflow’s response target. Keep the slow exceptions visible instead of reporting only the first model’s response time.
Evaluate routing errors separately from generation errors
Prepare cases with known expected fields and dispositions before adjusting the policy. Include straightforward requests, synthesis cases with complete evidence, missing attachments, conflicting records and requests outside scope. Keep a separate set for checking the policy after tuning it.
Compare a smaller-model-only path, a stronger-model-only path and the proposed routed path on matched inputs. Assess the final accepted brief and required review effort. Preserve cases where neither model succeeds; the stronger-model baseline is a comparator, not ground truth.
For each task, record the initial route, the reason for any escalation, each acceptance result and the final disposition. This exposes two different routing errors: spending on a stronger model when the smaller path was sufficient, and accepting a smaller-model result that should have been challenged. A single average escalation rate conceals that distinction.
If a learned router or answer scorer is used, evaluate its decisions on the actual task distribution. Preference labels, structural validation and verified factual correctness answer different questions. Changes in request mix can also change how often the route works and what it costs, even if neither underlying model changes.
Keep the fallback bounded and reviewable
Set a limit on repeated attempts and define the final destination when the stronger-model path also fails. In the fictional workflow, that might be an unresolved brief sent to the responsible employee with its missing facts and failed checks. Avoid a loop that keeps generating until one answer appears acceptable.
Use only approved processing paths for every model in the route. Adding another provider or endpoint requires the same data-handling and integration review as using it directly. Keep action permissions consistent across the route; a stronger model should not inherit broader operational authority merely because the first answer failed.
Our model-upgrade evaluation guide covers choosing and rolling out a changed setup. A routed workflow adds an ongoing policy decision for each task. Start with one bounded process, define its acceptance and escalation destinations, and compare the complete path before increasing the share of work sent through it.