The largest AI model is not the best for your customer service.

General AI benchmarks tell us very little about customer service

When choosing an AI model, the logic seems simple: pick the smartest one available, especially if it tops the well-known AI benchmarks. However, this logic does not always hold true for customer service.

Many popular AI benchmarks measure performance in areas like coding, mathematics, general knowledge, and complex reasoning. While this highlights a model's general capabilities, it says very little about how well it will perform as a customer service AI agent.

In reality, customers rarely phrase their questions perfectly. For example, they might say: "My order was supposed to arrive yesterday, but I haven't received anything yet. Can you check where it is?"

The AI agent must first understand the customer's intent. It then needs to look up the order, check its current status, determine what information it is allowed to share, and take the correct next step.

This means it is not just the final answer that needs to be correct, but the entire process.

Why we developed CM-ServiceBench

CM-ServiceBench tests AI models on complete customer service conversations. It evaluates more than just the final answer, assessing every step of the journey: reasoning, using the right tools, executing workflows correctly, and staying within guardrails.

We evaluate three key areas:

Choosing the right path

The first step is understanding what the customer actually wants. That sounds simple, until they express themselves in incomplete, ambiguous, or unexpected ways.

A good AI agent must still identify the correct intent and determine the next steps. A mistake at the start of the conversation can mean that even if every subsequent step is technically correct, the final outcome is still wrong.

Executing workflows correctly

Most customer inquiries require more than a single answer. Processing a return, rescheduling an appointment, or checking an order status involves multiple steps: identifying the customer, retrieving data, verifying eligibility, making a decision, and finally executing the action.

The order of these steps also matters. An agent that completes five out of six steps correctly can still produce the wrong outcome. This is what makes customer service fundamentally different from simply generating a good response. An AI agent must be able to follow and execute a process reliably.

Staying within guardrails

A good AI agent must also know what not to do. It should not hallucinate information, share internal instructions, expose data not meant for the customer, or take action when it lacks sufficient information.

For an AI agent in production, this is not an optional feature; it is a fundamental requirement. An agent is only viable if it is both helpful and predictable.

How do you evaluate a conversation?

With CM-ServiceBench, we have models handle hundreds of customer conversations across various scenarios and languages. A second AI acts as the customer, equipped with its own goal, context, and information, and conducts a conversation with the AI agent under test.

These scenarios are based on patterns found in real-world customer service processes. We then evaluate the entire conversation, not just the final response. Did the model choose the right path? Did it use the correct tools and actions? Were the steps executed in the right order? Was the information processed correctly? And did the conversation lead to the right outcome?

This distinction is crucial. A model can give a highly convincing response while executing the wrong process behind the scenes. To the customer, the interaction seems successful, until they realize their appointment was never rescheduled, the wrong information was retrieved, or they were routed to the wrong department.

Strict versus partial: When is a conversation truly successful?

A conversation can go mostly well but still contain an error along the way. That is why CM-ServiceBench measures results in two ways: partial and strict.

The partial metric measures how many individual parts of the conversation were executed correctly. The strict metric is more demanding: a conversation only counts as successful if the entire interaction is flawless.

This distinction is vital for customer service. A model that gets nine out of ten steps right can still leave the customer with the wrong result.

The leaderboard suddenly looks different

When tested this way, the rankings look quite different from what general AI benchmarks might suggest. In our tests, compact models from the GPT-5.6 family are actually among the top performers for these customer service tasks.

The most interesting insight is not just which model comes out on top. Compare GPT-5.6 Terra and Opus 4.6, for example. Opus 4.6 scores highly on partial but lower on strict. This means the model often gets far into the process but completes fewer conversations entirely correctly.

In our tests, GPT-5.6 Terra is much more consistent at finishing the job. For the customer, this is the only difference that matters. They do not care how impressive the reasoning was along the way, only whether their problem was actually resolved.

Bigger does not mean more reliable

This does not mean larger models are inferior. They are designed to handle a wide range of complex tasks. This extra capacity is highly valuable when an agent needs to make complex decisions, process extensive context, or resolve unpredictable issues.

However, not every customer service task requires this level of complexity. For many processes, quality is about consistently following instructions, using the right tools, and never skipping a step. In these cases, higher general intelligence does not automatically translate to a better outcome.

As a result, you might end up paying for extra model capacity without seeing any performance improvement for that specific task.

Speed is also a measure of quality

Furthermore, a benchmark score does not tell the whole story. Because AI agents interact with humans, response time directly impacts the user experience. If a model takes several seconds longer to respond to every message, the conversation quickly feels sluggish. This is even more critical for voice applications, where any delay is immediately noticeable.

That is why CM-ServiceBench evaluates response times alongside reliability.

CM-ServiceBench: performance versus speed

Each dot represents a model. The further to the left, the faster the model responds. The higher the dot, the more conversations were completed flawlessly.

The chart illustrates why there is no single, obvious winner. GPT-5.5 is the fastest, responding in an average of 1.8 seconds per message, but it does not achieve the highest strict score. At the other end of the spectrum, GPT-5.6 Luna with thinking scores highest in the benchmark at 71, but comes with a slower average response time of 3.2 seconds.

GPT-5.6 Terra sits in the middle, achieving a high strict score of 69 with an average response time of 2.8 seconds. This highlights what model selection is really about in practice: finding the best balance between quality and speed for a specific task, rather than just chasing the highest score.

There is also a third factor to consider: cost. A model that scores slightly higher but requires significantly more computing power and capacity is not necessarily the best choice for a process that runs thousands of times a day.

The best model depends on the task

Asking "what is the best AI model?" is too broad. A better question is: what is the best model for this specific agent and task?

A compact, fast model might be ideal for a predictable workflow. For a more complex agent that requires nuanced decision-making or processes large amounts of context, advanced reasoning capabilities add real value. Meanwhile, for processes where every second counts, a faster model may be the better option, even if another model scores slightly higher on strict accuracy.

This also means you do not have to standardize on a single model for your entire customer service operation. Different AI agents have different requirements. Even within a single agent, simple classification tasks might call for a different model than complex analysis or action execution.

The key takeaway from CM-ServiceBench is not that one specific model wins. Instead, it shows that you must evaluate models based on the actual work they will perform. While general benchmarks are interesting, you ultimately need to know which model runs your specific processes most reliably, at a speed and price point that make sense for your business.

Model selection in HALO

This is why we do not limit you to a single model in HALO. You can choose the right AI model for each agent, and even assign different models to specific tools. This allows you to deploy advanced model capabilities only where they add real value, rather than running every simple process through your largest, most expensive model by default.

These requirements will also continue to evolve. As new models emerge, existing ones improve, and performance, speed, and cost dynamics shift, CM-ServiceBench allows us to continuously test models against the same service tasks. This ensures your choices are always backed by up-to-date, real-world data.