Dialog-RSN-1: a voice model that hears calls the way humans do

Today, PolyAI is introducing Dialog-RSN-1, a dialog model that directly perceives the user's audio. Dialog-RSN-1 fuses turn taking, speech recognition, function calling, and response into a single audio-native model, bringing the contextual intelligence of an LLM to the full audio stack. It's already handling live calls in production, achieving the lowest latencies we have ever measured, and powering conversations that feel markedly more fluid and intelligent than cascaded systems.

For the enterprises that build on PolyAI, that means dialog agents that hear and handle a call the way a great human agent would: catching hesitation, hearing frustration before it escalates, and knowing exactly when to speak.

A new approach to hearing calls

There are broadly two established voice agent architectures: (1) using cascaded systems, or (2) using a speech-to-speech model. Each of these has its own compromises. Dialog-RSN-1 introduces a new architecture that avoids the drawbacks of the established techniques.

The traditional cascade has an inherent information bottleneck . The LLM at the heart of the voice agent only receives the single best guess of what the Automatic Speech Recognition (ASR) thought the user said. There is no audio signal reaching the LLM, no understanding of your tone, and any uncertainty of a potentially misrecognized word is lost. Furthermore, the ASR is usually a much smaller model, with a lot less context, and no ability to reason, meaning the LLM often has to handle misrecognitions.

To improve a cascaded system, you must improve its individual pieces: tweaking the end-pointing parameters, biasing the ASR towards your use case, adapting the ASR throughout the conversation, etc. You end up with a host of manually tweaked configuration parameters and hand-crafted patches, often balancing perceived latency and fluidity with robustness. And these parameters rarely generalise across use cases, needing to be separately optimized for different types of caller.

Speech-to-speech models address many of these issues, but they force you to give up control . Models like GPT Realtime and Gemini Live go fully end-to-end, removing the ASR transcription step. The LLM hears the audio, and produces audio in its output, but the output voice is baked into the model, with limited choice for how it sounds or control over how it pronounces terms. Does GPT realtime mispronounce your brand name? Tough luck! Speech-to-speech is also expensive, and that’s reflected in the cost of these APIs. Any general purpose model that operates at a time-scale sufficient to generate natural speech requires a lot of GPU resources to run. Some of these run full-duplex, keeping an always-on stream that listens and speaks at the same time; others operate turn by turn. The always-on variants are the priciest, pinning a GPU for the entire call.

In our testing, we are not convinced that many of the advertised speech-to-speech models are indeed using end-to-end models. GPT Realtime 2.1 scores on par with cascaded systems on audio-aware evaluations, suggesting it is just making educated guesses based on a transcription when it needs to reason over audio. It's likely these are just well-engineered cascaded systems.

Dialog-RSN-1 is audio-aware on the input side only. It fuses end-pointing, ASR, and response generation into a single audio-aware LLM, so it can reason directly over the raw audio signal, but it leaves speech generation to a separate TTS system. We believe the input side is where most of the benefit of audio-aware LLMs is. There are only limited cases where TTS generation needs to hear the user's voice, especially now that modern TTS systems are all promptable and support emotion tags. Keeping generation separate means we retain full control of the output voice.

Dialog-RSN-1 also runs as an ordinary request-based LLM, not a live audio stream. Rather than pinning a GPU and decoding every frame, it's probed on demand: a trigger-happy VAD and a few timers decide when to run it on the audio so far, and the first token of its reply settles whether it should start to speak. The cheap acoustic cues only pick when to probe; the model, with full context, makes the actual turn-taking call. So we achieve the fluid, audio-native feel of a speech-to-speech model, with the voice control and cost efficiency of a request-based system.

What Dialog-RSN-1 can do with the full call audio

A skilled human agent listens to the whole call. They hear the frustration of a customer struggling to remain in control, the hesitation before they say "yes", the line cutting in and out when you’re going through a tunnel: all of the messy parts of natural conversations. Dialog-RSN-1 works the same way.

These whole-call signals are what other models miss: cascades throw it away at the transcript, and speech-to-speech models keep it but don't reliably use it. In contrast, Dialog-RSN-1 hears the user's tone of voice, senses timing, and can pick up on other non-speech audio cues. It is able to combine the audio signal with the contextual information in the system prompt and conversation history to understand what you are saying better than any cascade.

The model can also be prompted on what it hears, like a caller's accent, mood, or the noise on the line. It uses the conversation and system prompt to sort out what a transcript gets wrong: a mispronounced word made obvious by context, a caller spelling "Matthew with two Ts", or a brand name like "Audibel" that isn't the word "audible". And it handles the audio that trips up cascaded systems, like spotting a human versus a machine, waiting for the beep, or holding through music.

Dialog-RSN-1 decides when to speak. It's trained to respond quickly when you have finished speaking, but not interrupt you during your pauses, reducing the trade-off between snappy turn-taking and talking over the user. And since the turn-taking is now owned by the LLM, it becomes yet another behavior that we can tweak in the system prompt:

Dialog-RSN-1 has massively simplified the configuration of our live deployments. Turn-taking is baked in (no silence windows or probability thresholds to tweak), and contextual speech recognition comes for free. The model naturally has all the context it needs from the conversation and system prompt, so it does well even for use-cases with tricky terminology such as in medical domains. No more turn-dependent ASR biasing, or swapping ASR models depending on the conversation state.

How we built it

We built Dialog-RSN-1 through further post-training of open weights multimodal models: supervised finetuning (SFT) and reinforcement finetuning (RFT) on in-house training datasets. Our training pipeline is broadly base-model agnostic, so we have evaluated a variety of architectures including Gemma, GPT-OSS, Qwen, and Mistral. We will share more results comparing base-model architectures in an upcoming technical report. We aim for sub-300ms latency when served on A100 GPUs, meaning we consider models in the 8B dense to 30B sparse range.

The first stage of our training, SFT, trains the model on a dataset of millions of conversations, based on real deployments and synthetic conversations, with expert teacher labels. This stage brings performance close to the general purpose API models, but the subsequent Reinforcement Finetuning pushes it higher still, refining the model's reasoning and instruction following.

Post-training teaches the base model to generate our custom output format, containing all the predictions we need: turn-taking decisions, efficient reasoning, text response, tool calls, knowledge citations, and speech transcriptions. Turn-taking is the very first token predicted by the model: one of EMPTY (no speech detected yet), ONGOING (user has started but not finished talking), or COMPLETE (the user turn is finished, and the assistant should reply). This is very fast to predict, since it is the very first output token, and the KV-cache is warmed up while the user is speaking.

Concretely, Dialog-RSN-1 is driven by a helper VAD that fires on two triggers: acoustic events and a regular time interval. It's tuned for high recall over precision, which is safe because a false trigger costs almost nothing. So the poll only has to propose reasonable timesteps; Dialog-RSN-1 makes the final call. A full-duplex model would drop the VAD and run continuously instead, decoding every incoming frame to keep a live turn-taking verdict, which pins a GPU for the whole call. Probing on demand gives the same responsiveness while leaving the GPU free between probes. That makes Dialog-RSN-1 far cheaper to run than an always-on model, with the compute cost and load profile of a normal request-based LLM.

Dialog-RSN-1 does transcribe the user's speech, but only after it has responded or called a function, and after grounding its action by citing its knowledge sources. Transcription happens last, in parallel to speech generation and the agentic loop, so it adds no latency the user can perceive. We don't strictly need the transcript, but it's useful in practice: for logging, context compaction, and text-based guardrails.

A lot of work has gone into making Dialog-RSN-1 fast. Beyond using a modestly-sized base model, we employ various tricks:

Measuring what happens on a real call

We're sharing results on Dialog-Eval, our benchmark for the breadth of capabilities an agent needs to actually run a customer call. We'll release it as an open-source evaluation framework in follow-up work. Dialog-Eval directly tests an agent's ability to get what it needs from the user or from the system, to deliver information back to the user, to act with real effect on its environment, to recognize when it can't help, and to recognize when it must not proceed.

We deliberately cover capabilities like understanding and reasoning over dates, times, characters, and strings, as well as following instructions, adhering to the conversational flows defined in the system prompt, and reliably calling tools. All examples represent the user side as audio, with a variety of accents, noise conditions, and audio quality, from clean speech to degraded phone lines.

Dialog-Eval is a point-in-time evaluation. Each example is a call truncated at a single decision point, drawn from real deployments and realistic synthetic conversations. Everything up to that moment is fixed: the system prompt, the conversation history, and the user's latest audio. We score only the model's next step, and that step is atomic, either one tool call or one user-facing reply, never both. One example stops right after the caller has read out four digits of a six-digit reference, and checks if the model keeps listening instead of replying early. Another stops as the user asks to move their appointment, and checks if it calls the booking tool with the right arguments rather than answering from memory. Because the context is fixed, a failure points at one specific targeted behavior.

In contrast, most benchmarks measure this with full conversation rollouts. A rollout doesn't control which behaviours you actually exercise. You run the whole conversation and hope the hard cases turn up. It also depends on a realistic user simulator, which is difficult to build and adds a lot of variance, so the same model can pass or fail the same scenario depending on what the simulated user happens to say.

Dialog-Eval

Figure 4 reports overall Quality score, and pulls out Audio and Safety scores, demonstrating some core comparisons across current models. We report both real-time capable models, which achieve reasonable latency for a voice call, and slower reasoning based audio-aware models such as Gemini Pro, to give an idea of the quality ceiling on this dataset. Dialog-RSN-1 achieves the highest scores among the real-time capable models.

These gains are already showing up in real enterprise customer conversations running on our platform. After moving to Dialog-RSN-1, a national restaurant group saw an 11% relative increase in containment (the share of calls resolved without handing off to a human).

The Audio score measures the model's performance on examples where the audio modality and conversational context is expected to be particularly helpful or indeed necessary, such as identifying a user's accent, tagging noise conditions, or understanding an obscure domain-specific term from contextual cues. In these cases, cascaded systems hit a ceiling of around 77: the optimum score achievable when the true signal is bottlenecked by the text representation.

Interestingly, GPT Realtime performs on-par with cascaded systems on Audio scores, suggesting that it may be in fact a well-engineered cascaded system, with limited or no ability to directly perceive the raw audio when formulating its response. (You can probe this for yourself by asking it if Americans pronounce the end of the alphabet as X Y Zed ).

The Safety score measures the model's resilience to jailbreaking and system prompt extraction. In these examples, the user attempts to make the agent leak its own configuration or misuse its own tools, and we evaluate the model's ability to respond appropriately and on-policy.

For each model, we present an estimation of its latency. This is computed on requests where the model outputs text for the user (as opposed to a tool call) from the moment a request was made to the model, to the moment the model returns enough text to start piping to TTS. For the cascaded models, this latency does not include additional overhead from speech recognition. And this does not factor in the fact that cascaded models typically need more conservative turn-taking/VAD thresholds, which adds perceived latency.

Dialog-RSN-1 reliably responds in under 300 milliseconds, with a much tighter distribution tail than models like GPT Realtime. Figure 5 shows the full distribution for Dialog-RSN-1 and GPT Realtime 2.1 in live deployments. And that speed carries into production: a large insurance provider cut response latency by 37% after switching to Dialog-RSN-1.

VAD evaluation

Figure 6 shows the turn-taking accuracy of Dialog-RSN-1 versus more conventional turn-taking models on a dataset of real user calls. Dialog-RSN-1 is the only model to maintain the full context of the conversation, and is orders of magnitude bigger, resulting in a dramatic improvement in accuracy versus these baselines. We include Gemini 3.1 Pro as an indicator of what a reasoning based model can achieve on this dataset; it is nowhere close to being fast enough for realtime (usually taking over 5 seconds to respond). Dialog-RSN-1 comes close to the performance of Gemini Pro on this task, especially for handling barge-in. Traditional systems often disable barge-in entirely to stay robust to noise, so the agent won't stop mid-sentence every time there's a cough or background chatter on the line. That keeps the agent talking, but it means the user can't interrupt. Dialog-RSN-1 doesn't have to make that trade.

WER Evaluation

Dialog-RSN-1 is evaluated alongside other contextual speech recognition models in Figure 7. It produces the most accurate transcriptions of any model we tested, even though we handed the others the same conversation context to work with. Dedicated ASR models normally transcribe independently of the surrounding conversation, seeing only the audio. Here we gave the strongest APIs the full context, either in the prompt or as extracted key terms, to make it a fair fight. It helps them: gpt-4o-transcribe's word error rate drops from 7.8% to 6.9% once it has the context.

Dialog-RSN-1 isn't a separate ASR model at all. The same joint model that responds also transcribes, already holding the system prompt, conversation history, and call audio. It answers first, then transcribes, so the transcript reflects the same understanding that produced the response.

The road ahead

For this first release, we focused on English, across both post-training and evaluation. The base model is inherently multilingual, and we've already seen Dialog-RSN-1 pick up non-English speech in production, so broader language support is next. Until then, Dialog-RSN-1 is our recommended model for English voice agents, and for rich web chat and non-English use cases, our existing Raven 3.5 model remains the best choice. We're working to fold both, along with all our other capabilities and modalities, into one joint model. More technical reports, and a paper introducing Dialog-Eval, are coming soon.

Until now, building a voice agent meant a trade-off. Cascaded systems flatten the call into a transcript and lose the audio, while speech-to-speech models keep the audio but give up control of the voice and the cost efficiency of a call. Dialog-RSN-1 gives up neither.

Hear the difference for yourself. Existing PolyAI customers can enable Dialog-RSN-1 in their projects today. New customers can request early access to be first in line when the public API opens.