TAO-1 for Contact Centers: Interleaved Reasoning and Low Latency

Hi, I'm Zexia from Retail. Today we're introducing TAO-1, which is the best model for contact center.

We've integrated with different model providers and mainstream LLMs for the response engine part.

And no matter if you're using cascading or duplex model, they all needed LLM.

LLM is the center of solving the complicated tasks and the nuanced situations.

For the contact center setting, a lot of times people just add more and more instructions to the system prompt to steer the model's behavior.

So these additional prompts makes the task harder and more difficult to follow instructions.

In terms of the duplex models, these models often perform delegations for the harder tasks to a powerful LLM in the background.

For example, when chatting with other duplex models, I would often hear, uh, "Wait a second, let me think," and then I hear a brief pause when the delegation is happening.

And across many turns, that sounds robotic.

That's why we propose a interleaved reasoning format where the model first answers, and then while it's answering, it's doing its internal reasoning, and then it will give the remaining of the response based on the reasoning.

This is the first productionized interleaved reasoning model, uh, for LLM, which is the best model for contact center.

With reasoning turned off, the model has to one-shot the answer.

It doesn't have any thinking beforehand to ground the reasoning.

With thinking, the model can eliminate the less suitable times and work out the answer step by step.

This is being able to execute the tool call at the right time and being able to, you know, solve all the complicated intention problems.

Like, for example, when you're calling, you might have someone who has like a thousand different intents and, and each intent has a defined SOP there.

So you want to follow that closely, and you want to make the right choice.

For example, a user could randomly ask, "What's the weather in Miami?"

or "What's a banana bread recipe?" The model, without additional prompting, is able to steer the user back at the task at hand and continue with the workflow.

There's a, uh, impossible triangle that people have been trying to aim for within the models is, you know, latency, cost, intelligence.

We've nailed all of them. A regular voice agent, the cost might not be the biggest factor, right?

But here, we're talking about like, you know, a million calls per month kind of volume for each customer, and this is where you need to consider the cost and how you scale here.

This model is highly intelligent for the contact center tasks, and latency is extremely low.

It's the time to first token you see is around 150 milliseconds, and you can hear the first audio at around like, you know, 200 to 250 milliseconds.

First is our custom, uh, inference stack.

Second is, uh, we skip the network latency by hosting the model directly in our production cluster.

It's more like a professional agent helping you solve your tasks or trying to sell you something or trying to, like, book you appointment, right?

Those will be different wording and stuff.

This is the go-to model if you want to ever go for any contact center use cases.