Can AI really hear a call the way a person does?
About the show
Hosted by Nikola Mrkšić, Co-founder and CEO of PolyAI, the Deep Learning with PolyAI podcast is the window into AI for CX leaders. We cut through hype in customer experience, support, and contact center AI — helping decision-makers understand what really matters.
Summary
PolyAI just launched Dialog-RSN-1, its first audio-native model, and Nikola Mrkšić sat down with its builder, Matt Henderson, VP of Research at PolyAI, to unpack why it's a game-changer for building voice agents.
Most voice AI either flattens a call into a transcript and loses the audio, or goes fully speech-to-speech and gives up control of the voice. Dialog-RSN-1 does neither. It hears the raw audio directly, decides when to speak, and keeps text-to-speech separate so the voice stays under your control — all in under 300 milliseconds.
Nikola and Matt get into what “audio-native” really means, why a cascade can't hear, how auto-reasoning keeps the model fast, and why it beats every other real-time model on quality and quickness.
Watch the full episode to see how PolyAI builds dialog agents that hear the whole call.
Key takeaways
- Audio-native is not speech-to-speech: Dialog-RSN-1 listens to raw audio to decide both how and when to respond, but outputs text and function calls, keeping text-to-speech separate. That preserves brand voice control and reliable tool calling — the two things full speech-to-speech models still compromise on.
- Auto-reasoning resolves the contradiction in the name: “Dialog” demands real time, “reasoning” normally means thousands of slow tokens. RL fine-tuning teaches the model to reason only when it pays off — about 20% of requests — so the median added latency from reasoning is zero, and the worst cases add roughly 300 milliseconds.
- Turn-taking is the model's first token: A fast acoustic VAD proposes pauses; Dialog-RSN-1's first generated token decides whether it should speak, with attention prefilled while the caller talks and speculative decoding accepting around four tokens ahead on average — landing median latency around 250–280 milliseconds.
- Benchmarks built for real calls: PolyAI is releasing DialogEval, an open benchmark of practical, targeted prompts. Dialog-RSN-1 leads all real-time models, with big slow reasoners like Gemini Pro as the ceiling — and an audio-understanding category where one “end-to-end” competitor scores no better than a cascade, suggesting it pushes speech through a text bottleneck.
-
[00:00:00] Nikola Mrkšić: Talking about reasoning, how does it do auto reasoning?
[00:00:02] Matt Henderson: So dialogue reason, there's a contention in the name. So dialogue means it needs to be real time. Otherwise, you can't have a dialogue. And then reasoning models, there's a massive breakthrough in LLMs, like, do reasoning. Just think for thousands of tokens. And any, like, big LLM you're using day to day is using reasoning, but it's slow. So how do you do both? And that the big thing we're doing is auto reasoning, where the model will only reason when it would help. So as part of our RL fine tuning that we do, we generate multiple candidates from the model, and we do some with reasoning, some without. And then we'd see, if you didn't do reasoning, do you do worse, basically?
[00:00:51] Nikola Mrkšić: Hello, everyone, and welcome to another episode of deep learning with PolyAI. Today, I'm here with Matt Henderson, our VP of Research, and we're here to talk about the release of our new model. And I'll just hand over to its creator.
[00:01:03] Matt Henderson: Thanks, Nicholas. Thanks for having me back on. Today, we're gonna talk about a new model dialogue, reason one. I think it should be quite a beefy topic for the podcast because it's our it's our first audio native model. So instead of having a cascaded system where you got speech recognition bolted on the front, LLM is then fed with just one best hypothesis typically for what the speech recognizer thought the user said. It's oftentimes that LLM just wishes it could hear what the audio was. Was that a mispronunciation? Like, are they talking to the same person? Is that background noise? Am I listening to music? Whatever. And that dialogue reason one can because it is a audio LLM. It's not only listening to the audio in order to decide how to respond, but it's also deciding when to respond. So we're getting that natural fluid turn taking that you get from speech to speech models, but we'll get into a bit later, I guess. It's not speech to speech full. We don't have a lot of the compromises that you have to sort of take if you're going full speech to speech. But I think we get all of the mostly all of the benefits. Right? So, like, true audio understanding, contextual speech understanding, super fast latency, and, like, fluid, promptable tone taking even.
[00:02:15] Nikola Mrkšić: Look, I think, you know, just we talk about dialogue. And twelve years ago, I think, was the first time I read a dialogue research paper by a man called Matt Henderson, and it was a dialogue state tracking challenge too. Also, fun fact, four people looking at the model name, as we looked at whether it should be the American or the British spelling, I think that, uh, your papers followed by mine were the ones using the American spelling most consistently. But I think it's you know, back then, the state tracking was about both, like, memory through turns and about the multiple speech hypotheses that we would then look at to kind of figure out what the model really said. I think it's, like, so in the spirit of deep learning that it's now just, like, been gobbled up into one mega model. And mega models are, like, the theme of every modality. Why not fully speech to speech? Why is it just an audio LLM?
[00:03:02] Matt Henderson: For it to say, like, yeah, I'm kind of excited that we are doing spoken language understanding again, and we're not just pretending that speech has been solved and feeding into text based LLM. And a lot of our work back then was doing stuff like that, like consuming the confusion network, for example, and that kind of thing. Why not speech to speech? You're like, well, it is speech in and still text out. So dialogue reason one is outputting either function calls, tool calls, so they go into the agentic loop, or it's like response in text to the user, which goes into text to speech. So we're keeping text to speech separate for a few reasons. I guess the first main one is controllability. So So if you're adopting a speech to speech model, you will be adopting their built in voices. So you get a handful of voices that you pick to represent your brand. If it mispronounces the name of your company or, like, one of the core things you sell, like tough luck, I don't know if you can, uh, reliably prompt them to pronounce things correctly. I don't think so. So, yeah, when you get controllability, you don't have to compromise there. I also feel like a lot of the motivation I like end to end models. I think that's where we're gonna go. That's where the industry is going. But the fact is that it's a pain to train those. Like, the data you need is just speech and speech. That's annoying. They're also just lagging. They've come a a long way, like, this calendar year. They're still a bit behind on instruction following calling tools reliably.
[00:04:23] Nikola Mrkšić: I think rather than just saying, like, it's not there yet, there's also maybe less to gain from tying that into the loop. We just did, like, a podcast episode, the previous one on word error rates where Oliver Scholson was kinda, like, showing me things that almost look anticorrelated performance of the end to end system with word error rate. It's not always anticorrelated, but the fact that it could even be anticorrelated shows you just, like, how much, a, it's not solved, b, it's not necessarily indicative. But, like, at least, you know, like, having that audio, it can reason in and goes like, hey. My name is Nikola Mariccik. Nikola is recognized. The blob after is weird, but then it goes and thinks, and it's like, hey. Something within each is probably coming because Nikola is the most common Serbian name. And, like, it actually works reliably. I mean, that would be reason one. So I'm astounded that it works. But, look, with text to speech, what would be, like, the main benefit of tying it in together?
[00:05:12] Matt Henderson: I was thinking is, like, oftentimes, it's kind of like cherry on the cake. It's not this, like, fundamental problem that you would get. If you add audio on the input side, then you fix a lot of problems. You remove the information bottleneck going through text. It can not get tripped up by mispronunciations or, like, weird new terms or rare names. On the output side, to me, it always feels like, well, okay, it could maybe match your tone or it could match your pronunciation. It's all, like, good stuff that would drive more human like natural conversations, but it's no killer use case, I don't think. So I like the direction, making everything end to end learnable and having gradients go from what the assistant said all the way back to what you said is, like, the ideal world. I guess there's a distinction within speech to speech that's worth mentioning as duplex versus more, like, turn taking stuff. With dialogue reason one, we have tried to get the benefits of, like, the duplex world with the sof, like
[00:06:10] Nikola Mrkšić: So maybe just for the audience, kinda, like, can you maybe just, like, distinguish between, like, the turn taking and duplex approaches?
[00:06:16] Matt Henderson: So duplex is basically the assistant can talk while it's listening, more or less. In practical terms, it means that you use the transformer to, like, have a constantly streaming channel, both for the users, like, input audio tokens and the audio tokens it's deciding to generate. So if you have a caller, you must assign that a GPA and keep it live. So you start to see this is a nice solution, but it's expensive solution. But what it allows you to do is to model anything. That as a probabilistic model, the system can backchannel while you're talking, say, mhmm, yeah, at the right times, hopefully. You can interrupt it whenever. It can interrupt you whenever. On the other hand, with, like, turn taking, you maybe have a more explicit state machine. Okay. The user's talking, and then maybe I will take a decision if I should interrupt to say fixed time step and that kind of thing.
[00:07:07] Nikola Mrkšić: Interrupt is more of a hardware interrupt, like stop, stop, stop, stop, stop. Right?
[00:07:11] Matt Henderson: Yeah. A duplex model doesn't even necessarily have a concept in its mind about interruption. It's just it would just be generating silence, and then it would generate something that's not silent. So I think that's where it's going. These models that are right there that are duplex, they're either, like, very toy, only capable of sort of chitchat.
[00:07:27] Nikola Mrkšić: So that's the d n v I forget the name.
[00:07:30] Matt Henderson: The PersonaPlex one. They came up with the one that can do function calling stuff, but it's very limited in its instruction following and ability to, like, do real but, you know, that's gonna catch up. So with dialogue based on how does it work, how does it do turn taking, we train it so the first token that outputs for any generation after a user audio is its decision of whether or not it should speak. So in under a hundred milliseconds because while they're speaking, we, like, presale its attention with everything up to what they've said. And then as they speak, we can chunk more in. And then we say, okay. We'll use a simple kind of, like, energy based acoustic VAD to say, here's a pause. The user's paused. Do you want to speak or not? And then dialogue reason one will say, yes. This is a good time for me to speak, or no, the user's still speaking, something like that. And then immediately after that first token, it just generates the rest of its response. So that's, like, effectively zero time to first token after that.
[00:08:26] Nikola Mrkšić: So it's almost like this VID is like some proto acoustic model of the old speech recognition, and everything else becomes one model.
[00:08:33] Matt Henderson: That's right. We use this acoustic video as a helper, like a dumb, small model. Well, so we pick the best one we can, but it's dumb in that. It's it's not an LLM. Right? It's not billions of parameters. It's not don't have the full context of the conversation. It's not, like, dynamically updating based on the context of what it's heard. All this stuff that dialogue reason one does. But it's fast and cheap to run, and it proposes, like, reasonable points in time for dialogue reason one to take turn taking decisions. So then the the big model acts as, like, the brain on top that's acting a lot less frequent time scale and then deciding how to like, when and and how to reply. That helper model's tuned to be super fast. So once you have finished speaking, it can respond super quickly. But you don't have that same compromise we would used to have with cascaded systems where if you pause for long, assistant would be too eager to interrupt you. Because it can hear you. It will just reject that pause so you know they're still talking.
[00:09:29] Nikola Mrkšić: What else did we do to get the latency down?
[00:09:31] Matt Henderson: So there's a, yeah, a whole bunch. And, like, I guess that's part of what we do is we're optimizing for that user perceived latency. We're not optimizing for the cost of us to serve it. We just really want to make the latency super fast. So, like, we don't care really about three p or, you know, multiple. We want to make it any call just signed super fast because if you don't have super fast latency, it won't sound natural. Right? So we do a bunch of things. So, like, quantization is a big one. The model is quantization of where training will make that no compromise in in quality. We also do some metrics I alluded to where, like, while the user is speaking, we can precompute a bunch of stuff. So that, like, context is prefilled. We do speculative decoding. Have you seen the speculative decoding?
[00:10:13] Nikola Mrkšić: I have not.
[00:10:14] Matt Henderson: This is, like, a cool thing where you basically have a small helper model that's, like, ahead on top of some of the hidden layers of the main model. It is asked to predict, say, like, seven tokens into the future. What do you think? What does the small model think the big model's gonna say? So normally, obviously, you generate one token at a time autoregressive, and then you kind of hit with this loop where you have to wait for that token to be generated to feed in. If the small model could play guess seven tokens ahead, then you would just batch those through, and you'd make use of the efficient batching on the GPU, and you speed things up.
[00:10:45] Nikola Mrkšić: How do you know it predicts the right ones?
[00:10:47] Matt Henderson: Well, we just train it.
[00:10:48] Nikola Mrkšić: So it's almost a distilled model of the bigger one for predicting the next few rather than the next very many. Right?
[00:10:54] Matt Henderson: Yeah. Exactly. It's distillations. We match, like, the KL between the big model and the small model. So we train this model to match the, you know, the greedy decoding predictions going ahead, sifting tokens into the future. And on average, in production calls, it's correct up to on average, four tokens are accepted. So while the big model is generating, it's actually, like, accepting, say, you know, on average, four tokens into the future. Part of that is just we're exploiting our specialization. We don't need to turn it to, like, predict really well ancient French poetry or something. It's like our function calling format or turn taking format. Dialog reason one that, like, cites knowledge citations at the end, and all the these are, like, formatting tokens it can get, but also just the reasoning itself and the and the output is predictable.
[00:11:39] Nikola Mrkšić: You're talking about reasoning. How does it do auto reasoning?
[00:11:42] Matt Henderson: So dialogue reason like, there's a contention in the name. Right? So dialogue means it needs to be real time. Otherwise, you can't have a dialogue. And then reason, like, okay, reasoning models, there's a massive breakthrough in LLMs, like, do reasoning. Just think for thousands of tokens. And any, like, big LLM you're using day to day is using reasoning, but it's slow. So how do you do both? And that, like, the big thing we're doing is auto reasoning, where the model would only reason when it would help. So as part of our RL fine tuning that we do, we generate multiple candidates from the model, and we do some with reasoning, some without. And then we just see, if you didn't do reasoning, do you do worse, basically?
[00:12:23] Nikola Mrkšić: Uh-huh. So in training, you kinda look around somewhat without, and then, like, a wrong payment decides on its own whether it should be reasoning or not?
[00:12:30] Matt Henderson: Yeah. Using those rollouts during training that we we would then train that decision token. Should I do reasoning or not based on, like, the payoff? And in practical terms, like, 20% of requests get reasoning. And then when it does reason, we train that to be efficient and short. Like, let's say, it generates 50 tokens that might, in reality, be, like, 12 chunks of tokens that get accepted. But, yeah, we have, like, a lost term inside our reinforcement functioning for reducing the length. Basically, we optimize in reinforcement functioning for the added latency. So a p 50, a median added latency from doing reasoning is zero. Most cases, it reason. So then, like, EAT is where it, like, has Three hundred milliseconds or something. So if at some point in the call, when it gets to that difficult point, and it wouldn't do this twice inside one agentic loop, but just do it once, It would add, like, three 100 milliseconds, which I don't know if you can perceive. Just about perceivable.
[00:13:28] Nikola Mrkšić: It is perceivable, but, like, only if you want to be very scientific about it. Right? I think in practice, it's if it's really 300 and no more, then it's kind of immaterial for a real conversation. I mean, round trip times in about telephony setup can cost you more than that, potentially. That's really interesting. And, you know, I kinda like just putting, like, a bit of a marketing veneer on it. You could really say that it's the best dialogue model that can reason in under three hundred milliseconds or maybe, like, point five if you put it all together. Right?
[00:13:56] Matt Henderson: That way of framing it. Like, we're not saying we've trained the best reasoning model, but we're saying this is the best one that's reasoning in real time.
[00:14:03] Nikola Mrkšić: The best real time reasoning model, like, in the world right now?
[00:14:07] Matt Henderson: Yeah. For getting stuff done in a voice call. And I'll just, like, chat about latency and stuff. I would recommend listeners to check out the blog post, which should be out when this is released. We got some comparison versus, like, g t real time, for example, where our medium latency, we're aiming for under three hundred milliseconds. It's in, like, two eighty, two fifty or something.
[00:14:25] Nikola Mrkšić: Okay. Well, talking about Evas then, how did we evaluate, like, its performance other than the latency piece where we were kinda, like, pretty uncompromising around, like, this limit?
[00:14:34] Matt Henderson: We are releasing our internal benchmark soon. Um, I think it's gonna be called Dialog eval, and there are results from sort of early version of that in the blog post. In there, you'll see that Dialog reason one is performing the best amongst the real time models. We also show, you know, the big reasoning slow models like Gemini Pro as an idea for the ceiling. Like, what if you were allowed to think for a minute before he replied on this dataset, what would you get?
[00:15:02] Nikola Mrkšić: What was Gemini Pro chosen as the ceiling versus, you know, a kind of drop off model? Or
[00:15:06] Matt Henderson: The Gemini Pro is one that supports audio in the input side, unlike the Claude ones. So it's a nice model in that, uh, you can feed it the composition the same way you can dial a reason one.
[00:15:17] Nikola Mrkšić: Are there any other foundational ones which would be, like, contenders for that, like, all I'm judging?
[00:15:22] Matt Henderson: We have the GPT real time models and then the various, like, open source models as well, which would require post training to be competitive. We've taken open waste model and post trained it to do this. So all these open waste models, they can hear and they can respond. Just, uh, they don't have this turn taking capability, and they don't have very reliable function calling.
[00:15:45] Nikola Mrkšić: I mean, like, function calling really is where d s to s models also fall over right now. Right?
[00:15:50] Matt Henderson: Was definitely the case six months ago, and they're getting better. But it is kind of a difficult task to mix speech with function calling, I think, and that it can be quite hard, I think, for these models to understand what agent framework they're in. Like, what happens if I call a function? Do I need to say something before I call a function? Will I get an a chance to call another function? If I say something before I call this function, is that it? I'm not gonna be allowed to. So, like, part of the reason that we train the model is that all of that is just built into the weights because we train it to work in our agent framework where we have answers to all of those questions. The model would become allergic to, you know, speaking before calling a function because if you speak in our framework, you we're not allowed to call a function after.
[00:16:32] Nikola Mrkšić: It's fascinating. Okay. Like, maybe two two r e vowels and kinda, like, things we measure?
[00:16:36] Matt Henderson: So one thing we pull out is this audio category. So those are examples in the dataset where audio should help. So things like identify the background noise conditions to, like, in the context, there's a menu for a restaurant and the user's, like, mispronounced one of the things. So the first one, you definitely need to hear the audio to know the background noise. The second one, like, it should certainly help to hear the audio and to have the context. Those types of examples, that's where you see the big jump. So I guess the point is that having an audio native model, like, not only improves the speech understanding, but also opens up new use cases. So you can prompt it in new ways that you wouldn't have before, and that's what's measured in this audio category. So you could prompt it to be, like, if the user is somewhere noisy, then tell them you're gonna do your best, but, like, suggest that they move somewhere quieter if possible. Or if it sounds like the user is struggling to get their customer number, you say, okay. We could also look you up by your postcode or something. Interrupt the user and say that, like, it's like new use cases that the cascaded, they should be sort of random or educated guessing level. We actually do see that GPT real time on that category is about as the same as cascaded, you know, pipeline systems. So, uh, there's there's a bit more looking into, but to me, it suggests that this is a, you know, fancy pipeline system, like a well engineered ASRLMTTS rather than a true speech to speech model as advertised.
[00:18:02] Nikola Mrkšić: Yeah. I mean, it's interesting. I think that kinda, like, that surfacing out kinda, like, puts into question, like, there's a whole, like, vibe feeling of a demo where we do one thing versus, like, what's really under the hood and, like, how important that actually is. Because do you think it's more likely that it is, like, a semi cascaded system or rather, like, a safeguarded system that might have, like, cascaded bits intervening? Kinda like RVAD is there to provide, like, a signal.
[00:18:27] Matt Henderson: I think there's pretty strong evidence from what I've seen that is going through, like, a text bottleneck between speech and an LLM part. I had that sort of, um, demo I post on LinkedIn a while ago where if you ask it, do Americans pronounce the end of the alphabet x y zed? Then it responds, yes. Americans respond the end of the alphabet as x y z, which is what you would get if it had been transcribed as just the letter x y in a character. So, yeah, I mean, that that is that's one anecdotal example. But, obviously, on our dialogue about benchmark, we're gonna be releasing soon. Uh, you can see for yourself, I suppose, and evaluate whatever models you like.
[00:19:04] Nikola Mrkšić: Why do we not use other things? You know, I think that you know, there are things like the TelBench that people talk about as, like, benchmarks for this. What was wrong with them?
[00:19:12] Matt Henderson: I think well, yeah, TelBench has a couple. It has the original sort of text based one and the more recent voice version. And the text one is like, okay. What would happen if, uh, Fable was allowed to run banking customer call? Okay. It's interesting, but it's not a practical. The latency there would be crazy. And in reality, you know, we might use something like Fable to build the agent that talks to customers of a bank, but we're not going to use Fable to actually run, you know, thousands of calls a day with any with the customer. The voice one, it has unrealistic prompting, and that I think it's a good thing to measure still and, like, how well do these models do with quite open prompts and general tools that they're allowed to call. But it's not a targeted evaluation of can an LLM do all the things it needs to to make a customer call work. It relies a bunch on the user simulator being accurate, which is not really, where there's a lot of variability there. And, yeah, I think if you were to build a practical system, you'd not prompt it in the way that they have in in this evaluation. So we're interested in that benchmark, but we wanted to build dialogue of Vowel so that has some realistic, practical kind of prompts that you would see. And it's targeted in that we're not relying on the user simulator, but we're just putting you in all of the balanced context you would find yourself in if you're an ALM given this type of task of targeted evaluation of those capabilities.
[00:20:39] Nikola Mrkšić: Okay. Well, look. I mean, I think evaluating dialogue systems has always been hard. I think we're releasing that. We're gonna do our best to give the community something that they can use as a more pragmatic real proxy for whether they've got a good model or not. I think the more exciting bit is also we've got that one in production in the platform right now. We've got it in the hands of a few select partners that are already kinda putting it inside their platforms as well. It will be released as an API in the coming weeks. And if you're interested in trying it out early, you can get in touch with us and get into the early access plan as well, working for feedback and for just usage because it's something we worked on for a long time. We're really excited for the world to experience both in, like, using our platform and using other platforms.
[00:21:25] Matt Henderson: Super excited for more people to get their hands on it.
[00:21:27] Nikola Mrkšić: Yeah. Well, Matt, thank you for today. More importantly, thank you for leading this work and getting us here. And as always, guys, please like, share, subscribe, and we'll see you in the next one.
[00:21:37] Matt Henderson: Bye. End of turn. End of turn.