The linguist teaching enterprise AI how people actually talk

Listen to enough automated phone calls and the same small failure repeats. The caller waits a beat, hears a tidy, slightly over-helpful greeting, and, within seconds, is smashing zero and asking for a human. Nine times out of ten, the agent answered correctly. It just sounded like it was reading from a brochure.

Oliver Shoulson has built his career on the gap between being technically correct and being easy to talk to. As Agent Design and Engineering Lead at PolyAI, he designs the conversations enterprises put in front of millions of callers, and his rule cuts against the common instinct: a good agent doesn’t need to fool anyone. It needs to remove the friction that makes people work too hard.

On a recent episode of the Digital Project Manager podcast , Oliver revealed what that takes, from the milliseconds of turn-taking to the quiet damage done by an over-helpful “ LLM voice .” Here’s what stood out, and why it matters for any team putting an AI agent on the phone.

The linguist behind the voice

Oliver came to conversation design through language itself . At Yale, he studied syntactic variation, the small differences in how speakers build and arrange sentences inside the otherwise fixed rules of grammar. Outside work, he’s a visual artist, and both show up in how he approaches the job.

A linguist’s habit, as he explains, is to interrogate his own intuition. One sentence sounds right, another sounds off, and the work is to figure out why, then write the rule precisely enough that a machine could, in theory, follow it. That is close to what any good designer does. It is also why he believes the discipline doesn’t require a linguistics degree, just a willingness to notice what makes an interaction feel awkward and to ask what is really going on underneath.

No one gets gold stars for tricking anyone

Ask Oliver about the goal of conversation design, and he won’t say realism. “No one’s getting gold stars for tricking anyone,” as he puts it. The aim is to free up the caller’s problem-solving brain for the actual problem, instead of spending it on how to navigate the system.

He has a sharp example. Call a company, hit a phone menu, and you start reverse-engineering how the designer filed your issue. Is it an “account problem” or a “billing question”? You burn attention decoding someone else’s menu logic before you’ve touched your actual issue. Good design hands that effort back, so people fall into the conversational instincts they already have and spend their focus where it belongs.

He’s quick to add that natural doesn’t mean a free-for-all. Even an AI agent should be able to set boundaries with an abusive caller. The aim is to support good patterns of conversation, not to copy every human one.

Why “helpful” AI often sounds patronizing

Large language models are trained to be safe, agreeable, and thorough. Dropped into a phone call, those same habits curdle. The model explains things nobody asked it to explain, narrates steps the caller already knows, and over-confirms at every turn.

Oliver points to the linguistics of it. In cooperative conversation, we assume the other person gives us as much information as we need and no more. Say more than the task requires and you imply something extra. Tell an adult to let you know once they’ve clicked a button they obviously know how to click, and you’ve quietly implied they might not. Open a simple request with a six-line preamble about why you need an account number, and you sound like something standing far away rather than a partner working the problem alongside them.

That gap between over-informative and designed-for-voice is where a lot of agents fail:

When an agent trusts the caller with the context they both already share, the exchange starts to feel collaborative instead of scripted.

The details people notice without noticing

Spoken conversation is messy and fast. People interrupt, correct themselves, and talk over each other, usually with no more than a few hundred milliseconds between turns. An agent that takes three seconds to respond breaks the rhythm, and the caller feels it before they can name why.

Some of the hardest problems lie in the details that most people never consciously notice. Early in his time at PolyAI, Oliver tackled one around phone numbers. People give a US number in chunks, area code, then three digits, then four, and they expect a small acknowledgment after each chunk. Stay silent after “one two three” and the caller stalls, unsure you’re still there. Worse, the agent might decide three digits was the whole number. The fix was teaching the model to give that back-channel confirmation at the exact moment human conversation expects it.

Turn-taking, interruptions, pacing, the little “mm-hm” that says keep going: these are subconscious for callers and genuinely hard to build. Get them wrong, and the conversation feels off. Get them right, and nobody notices, and that shows up where it counts: more calls resolved without a human, and customers who hang up satisfied.

The deeper read: Oliver goes further on designing agents for real-world conditions in our webinar, Building AI agents for the real world .

Giving Selma a million voices

The payoff shows up in the work. Fogo de Chão , the Brazilian steakhouse chain, wanted its phone experience to carry the same hospitality as its dining rooms. So PolyAI cloned the voice of Selma, a longtime leader on its customer experience team, and built her into the agent that now answers calls about reservations and rewards. One person who had spent decades shaping how the brand sounds, given the range to greet far more guests than she ever could at once. The Selma agent achieves 95% customer satisfaction and completes 88% of bookings.

That kind of branded voice is something text can’t replicate, and it speaks to why people still pick up the phone at all. Oliver points to a decades-old idea from human-computer interaction called social presence , the sense of being with someone in real time. Research links a stronger sense of presence to higher trust, better adherence to advice, and more confidence that personal information is being handled with care. A well-designed voice agent can create that feeling, where a chatbot rarely does.

Automation that gives teams their time back

The upside for human teams is just as real. After bringing PolyAI on, Carter’s was finally able to give its contact center staff Black Friday off, on top of Thanksgiving. Most callers phone in about the same handful of routine questions, so letting the agent carry that volume frees people for the conversations that genuinely need a human. Oliver frames his work as supplementing human teams, not replacing them: the AI takes the repetitive load, and the hardest, most human calls still reach a person.

Adding more to make AI sound human, more warmth, more explanation, more polish, tends to do the opposite. Oliver’s work runs the other way. Strip out the friction, trust the context people already bring, and let them spend their attention on the thing they called about. There’s no trick in it, just good conversation, designed by someone who pays very close attention to how people actually have it.

Meet Oliver. Oliver Shoulson is Agent Design and Engineering Lead at PolyAI, where he builds the conversational AI that powers natural customer interactions for global enterprises. You can follow his work on LinkedIn .

Make every customer feel heard. Instantly. Speak to our team today.