Interactive benchmarks: evaluating model routing for AI agents

We founded Not Diamond on an early belief that intelligent routing would become the most important infrastructure layer in the multi-model future. Over the past three years, we have led the category by releasing multiple state-of-the-art routing techniques. Most recently, we pioneered long-horizon, cache-aware routing for coding agents.

In this document, we walk through our benchmarking methodology for routing and share how we provide tooling to our clients to benchmark Not Diamond by themselves.

Our approach to routing

Model routing is both a difficult research problem and a moving target as new models are constantly released. For long-running agent workloads, the interdependent variables and downstream consequences of model selection make the problem extremely challenging to solve well. While conceptually simple solutions abound, such as single-turn and first-turn routing, in practice these techniques are brittle and ineffective:

Single-turn routing: Assesses an individual request and predicts which model is best-suited to respond at the lowest cost. This approach will naively invalidate the KV cache on each turn of an agent session, costing more money than any individual model at lower performance.
First-turn routing: Uses the first message in a session to pin which model to use for the entire session. This approach is brittle to mid-session task and complexity shifts, which are common in long-horizon and interactive settings.

In contrast, Not Diamond Code is based on a novel routing technique that makes recommendations by predicting future rewards and costs for a given model and reasoning effort at each step of an agent sequence in a cache-aware manner. It takes into account current and previous session states, message and token counts, task complexity, the state of the KV cache, and intermediate and final reward signals to make effective model and reasoning effort recommendations over the course of an agent’s entire trajectory. Our approach is Pareto-dominant on real-world coding tasks and achieves frontier quality at a 20-80% cost reduction.

How to benchmark a router

Intelligent model routing aims to maintain frontier quality while reducing cost in real-world settings. As such, evaluations must represent how developers actually use agents in practice.

The dynamics of real-world coding agent usage are not captured well by static industry benchmarks, which focus on a single initial task description, run autonomously for a small number of turns, and are designed to test the upper limit of model capabilities rather than cost efficiency over varied workloads. Static benchmarks fail to represent the most important factors for real-world coding agent costs, in particular long session lengths and variable cache dynamics:

Long sessions: Coding agents accumulate growing context over long horizons, meaning that every input and output is reconsumed on every subsequent turn throughout a session, adding significant cost even when accounting for cache efficiencies.
Cache expiration: When enough time passes between user interactions, the cache can expire and the same context becomes dramatically more expensive to process, resulting in a 10-20x increase in input token cost over the entire context window.

For these reasons, we have invested in evaluation infrastructure for interactive benchmarks with simulated user agents which represent how developers actually interact with coding agents. Interactive benchmarks allow us to simulate long sessions and varying user response times, which makes them more useful for evaluating real-world routing economics.

Benchmarking Not Diamond with Harbor

Harbor is a widely used framework for evaluating agents and models, developed by the authors of Terminal Bench. We have worked with the Harbor team to add support for simulated user interactive benchmarks, a long-requested feature from the evaluations community.

To help Not Diamond users evaluate the router, we provide a benchmark repository that uses Harbor to run Not Diamond against baseline models on three interactive coding-agent benchmarks: SWE-Interact, SWE-Together, and Dialogue SWE-Bench-Pro. For Dialogue SWE-Bench, we use SWE-Bench Pro as the basis rather than SWE-Bench Verified, which has leaked into model training data.

The current version of our router runs in Claude Code with Opus 5, Sonnet 5, and Haiku 4.5 and targets Opus 5 xhigh quality at a 20%+ cost reduction. We have initially focused on routing between Anthropic models as they represent the greatest source of enterprise spend today. This has allowed us to validate our algorithmic techniques within the single-provider routing setting, and we are now rapidly expanding to other frontier and open-source models and harnesses to offer enterprises greater cost-efficiency and flexibility.

Below we report our benchmark results. On all three benchmarks, Not Diamond is Pareto dominant, approximating or exceeding Opus xhigh quality at a 20-80% cost reduction. Equally importantly, Not Diamond remains Pareto dominant even as task complexity distributions shift. In the second and third benchmarks, Opus class models outperform Sonnet models, whereas in the first benchmark the opposite is true. Our analysis shows that this is due to Sonnet’s reduced verbosity on long-horizon moderate-complexity tasks. In all cases however Not Diamond remains dominant.

Accuracy-versus-cost scatter plot for SWE-Interact comparing Not Diamond Code with Claude Sonnet 5 and Opus 5 configurations. Not Diamond Code scores 50.7% at a cost of $4.28 per attempt. — In SWE-Interact, Not Diamond Code achieves a score of 50.7% at a cost of $4.28 per attempt.

Accuracy-versus-cost scatter plot for SWE-Together comparing Not Diamond Code with Claude Sonnet 5 and Opus 5 configurations. Not Diamond Code scores 67.6% at a cost of $4.16 per attempt. — In SWE-Together, Not Diamond Code achieves a score of 67.6% at a $4.16 cost per attempt.

Accuracy-versus-cost scatter plot for Dialogue SWE-Bench-Pro comparing Not Diamond Code with Claude Sonnet 5 and Opus 5 configurations. Not Diamond Code scores 57.6% at a cost of $3.00 per attempt. — In Dialogue SWE-Bench-Pro, Not Diamond Code achieves a score of 57.6% at a cost of $3.00 per attempt.

Roadmap: model, data and harness diversity

While Not Diamond achieves state-of-the-art results within the single-provider routing setting, our roadmap is focused on several key areas where we will continue to drive performance improvements.

Model diversity

We have focused on routing between Anthropic models because they currently represent the greatest source of usage for coding agents across enterprises. However, we are now expanding model diversity to more open-source and closed-source models in order to improve product performance, cost-efficiency, and flexibility. In internal evaluations on Poly-SWE-Bench, we observed that incorporating GLM 5.2 and DeepSeek V4 Flash into our router improved accuracy by 3.6% and increased cost savings from 39% to 66% relative to Opus xhigh:

Accuracy-versus-cost scatter plot for SWE-PolyBench-Verified comparing an Anthropic-only model router, an Anthropic and open-source model router, and Opus 4.8 xhigh. Adding open-source models improves routing accuracy by 3.6% and increases cost savings from 39% to 66% relative to Opus 4.8 xhigh. — Adding open-weight models improves accuracy and cost savings.

We are working to add more support for open weight models and also to assess the performance of our routing algorithm over OpenAI models and other closed-source models.

Data diversity

Another important lever for improving routing performance is the quantity and diversity of the data we process for training. Through Monte Carlo simulations, we approximate an oracle router that perfectly selects models throughout an agent’s trajectory to maximize accuracy while minimizing cost. We find that not only is there significant headroom on our current Anthropic-only router’s performance, but also that the more data we provide during training the closer we approximate the oracle router’s performance:

Accuracy-versus-cost scatter plot for Dialogue SWE-Bench-Pro showing how additional training data moves Not Diamond from a low-data router toward the higher-accuracy, lower-cost performance of an oracle model router. — There is significant headroom on our current Anthropic-only router's performance.

We are working to expand our training data volume and diversity. This is important both for coding agent performance and also for expanding Not Diamond to non-coding settings such as knowledge agents.

Harness diversity

Harness diversity is another important lever for improving both user experience and performance. While today we support Claude Code, Not Diamond is designed to be harness agnostic and we are working to rapidly expand our harness coverage. Our architecture provides developers within enterprises the freedom to change harnesses over time while giving leadership a unified control plane for model routing cost management. Additionally, industry and academic reports have shown that some harnesses are more cost-efficient than others, offering additional opportunities to help enterprises save costs on their agentic workloads.

Expanding beyond coding agents

To date we have focused exclusively on coding agents at Not Diamond. This allows us to deliver exceptional quality in a critical domain with broad economic impact across the industry. However, our approach is applicable to any agentic setting, and in the coming year we expect knowledge agent work to overtake the inference volume of coding agent work. Beyond this, the greatest source of inference spend will ultimately come from background agents which are handling tasks autonomously, monitoring systems, and communicating with other agents. We plan to expand coverage to these domains over the coming year to provide enterprises with a unified routing platform across all their AI usage.

Conclusion

Not Diamond delivers frontier-level quality at significantly lower cost by routing dynamically across models throughout an agent’s trajectory. As we continue developing the product, we will further improve routing performance and savings, giving enterprises and developers a flexible, unified means to optimize their agent inference spend. Over time, we seek to expand from model selection to model optimization, enabling enterprises to own more and more of the intelligence layer and helping catalyze the multi-model future.

To request access to Not Diamond Code, please fill out this form or book time with our team.

Interactive benchmarks: a new methodology for evaluating model routing