IBM Research Tests Agent Routing On Cost, Latency And Accuracy

Model routing for enterprise AI agents is becoming a systems-cost problem, not just a choice between large and small models.

Hugging Face published an IBM Research post by Yara Rizk, Eyal Shnarch, Jason Tsay and Merve Unuvar that argued routers must weigh cache behaviour, workload shape, latency and governance before deciding which model handles a task.

The routing layer in the post has to account for infrastructure and execution conditions while the agent is running.

A classifier that sends easy work to a cheaper model and harder work to a stronger one would miss part of the system cost that appears after a task starts.

AppWorld Costs Reversed The Sticker-Price Assumption

IBM Research said in the Hugging Face post that Sonnet cost $79 in total, or $0.19 per task, across 417 AppWorld Test Challenge tasks using the same CodeAct agent, while GPT-4.1 cost $155, or $0.37 per task.

IBM Research also said a later latency-focused router example reached 84% accuracy for $93 and 83s, with a 21% cost reduction and 9% latency reduction compared with Opus alone and a 4% accuracy drop.

Independent benchmark validation sits outside the public record for either comparison.

Caching changed that bill.

Caching supplied the operating explanation.

Agent workloads can reuse large parts of the same context across steps, and the authors attributed Sonnet's advantage to lower cache-read pricing that benefited from that pattern.

Companies deploying multi-step agents therefore have to compare the cost of the full workflow rather than the nominal rate for a single prompt.

The cost comparison therefore sits inside the workload, not the model brand.

A router that looks only at a price sheet can choose the cheaper-looking model while missing cache-read economics, repeated context and the number of intermediate tool calls that determine the final bill.

Routing Difficulty Appears After Execution Starts

The authors argued that routing by task difficulty breaks down when the workload looks simple at the front door but expands during execution.

A contract-summary request, for example, can trigger retrieval, compliance checks, tool calls and rounds of revision, while a technical prompt may be handled efficiently by a specialised smaller model.

The production section names five routing criteria: cost, quality, latency, compliance and reliability.

It also lists enterprise constraints such as data residency, privacy rules and approved-model lists as conditions that can change the model choice for a task.

Latency adds another layer.

The authors identified routing overhead, hardware placement, endpoint load and cache warmth as variables that can dominate what a user experiences.

Routing once per task limits overhead, while routing at every step gives more flexibility but adds more decision points and operational complexity.

These limits make the routing decision more like capacity planning than prompt triage.

The model is one input, but the serving endpoint, cache state, approved-model list and expected tool path can all change the answer before the user sees a response.

The mechanism is especially relevant for agents that call tools or retrieve documents over several steps.

The Hugging Face post says each extra step can change cache use, endpoint load and governance checks, so the first routing decision may not describe the whole workload.

IBM Research Tests An Optimisation-Based Router

The router section moved model choice away from classification and towards optimisation across cost, quality and latency.

IBM Research framed that optimiser as a way to search the trade-off space rather than rely on a single difficulty score.

The same section put optimisation overhead at roughly 6 ms and 2 kB of memory per task, limiting the risk that the router itself becomes a bottleneck.

A standard difficulty-based router landed in a similar accuracy range at higher cost, according to the comparison, because it did not search the broader trade-off space.

The deployment question for AI teams is whether a router can keep that trade-off stable once the workload leaves a benchmark and enters governed production.

IBM Research wrote that more technical detail would come in a follow-up post.

The router's full technical design, underlying configuration table and customer deployment results remain outside the public record.