Why we built Evaluate
HALO has long supported manual conversation testing. In the test center, you ask a question and follow how your agents handle it. You can see which sources they use and whether they successfully perform an action through a connected system. This helps you understand what’s happening and identify errors.
When your agents handle large volumes of conversations, you need to test more situations than you can work through manually. Ten successful test conversations give you a limited view when you’re processing hundreds of thousands of interactions a month.
As the person building an agent, you also tend to test the conversation you have in mind. You know what information is needed and phrase your question accordingly. Your customers approach things in their own way. A voice agent might ask someone to speak in a full sentence and receive a single word in response. The conversation still needs to move forward.
We built Evaluate to make that testing scalable. It lets you check, across a broader range of scenarios, whether your agents do what you expect before you publish them.
What is Evaluate?
Evaluate is the feature in HALO that automatically runs collections of test scenarios. These individual tests are called evaluations, or evals for short.
For each evaluation, you define the customer question or situation you want to test and the desired outcome. For example, a customer might want to register a return, report a change of address or adjust their monthly energy payment.
For these tasks, the agent needs to achieve the intended outcome. A well-written answer is only one part of the conversation. If a customer wants to make a change, you also want to check that the agent completes that task successfully.
A collection of evaluations lets you test multiple tasks and conversation paths, giving you a broader picture of how your agents perform.
How does Evaluate work?
1. Create your test scenarios
You create a collection of situations you want to test. What questions do your agents receive? What tasks should they perform? And what counts as a successful outcome?
You can build the collection yourself or use Ask HALO to generate scenarios for you. You can also ask Ask HALO to suggest variations on a conversation you’ve already tested manually. This extends your testing to other ways customers might ask the same question.
The size of your collection depends on your use case. A setup with multiple agents handling a range of tasks needs more scenarios than an agent with one clearly defined task. In the episode, Rutger mentions collections of fifty or a hundred scenarios as examples.
2. Run scenarios multiple times
AI can respond differently from one run to the next. Evaluate therefore runs each scenario multiple times to check whether the agent consistently achieves the desired outcome.
HALO marks a specific test path as successful when every repetition passes. The pass rate for that path is then 100%. That percentage applies to the tests performed. How well your collection covers real customer conversations depends on the scenarios you include.
3. Review the results before publishing
After testing, you review which scenarios pass and which need attention. You can then make adjustments and run the tests again.
This is particularly useful when you change an existing agent. You want to know whether the change solves the intended problem and how the rest of your setup performs afterwards. Running your broader collection again can also reveal whether a change affects tasks that previously worked well.
Your changes stay in HALO Studio until you click Publish. Until then, your existing setup continues to handle customer conversations. You decide when to publish the new version, using the test results to inform that decision.
From an improvement opportunity to a tested change
Evaluate works alongside Analytics, Optimize and Ask HALO. Together, they support the process of improving your agents.
Analytics and Optimize help you identify where conversations get stuck or where opportunities are being missed. You might discover that a particular type of question is regularly handled poorly. You can then use Ask HALO to make an adjustment.
With Evaluate, you check whether that adjustment delivers the desired outcome. You test the affected scenario and run your other tests again. You then assess whether the change is ready to publish.
After publication, new customer conversations provide insights for the next improvement. You stay involved in the decisions, with support for analysing, updating and testing your agents.
Test your own scenarios when switching AI models
Evaluations are also useful when you’re considering a different AI model. A model might score well in a general benchmark, but you want to know how it performs on your customer questions and tasks.
Running your own collection again lets you test the model within your agent setup. This gives you a concrete basis for assessing whether switching models would be an improvement.
Discover Evaluate in HALO
With Evaluate, testing your AI agents becomes part of every significant change. You define what an agent needs to do, test it at scale and use the results to decide what to publish.
