Every voice AI agent has a specific job to do: paying a bill, booking an appointment, tracking a parcel. Yet about 50% of what callers say has nothing to do with that job.
They use pleasantries, ask for clarification, for you to repeat yourself, to hold on a minute while I put the dog out, and so on. And they come up in every conversation, regardless of the use case.
These are global behaviours. No matter what your use case is, your voice AI agent needs to cater for them. Here’s a full list of them all and how to define them.
Primary conversation handlers
We call these global behaviours primary conversation handlers. The thinking has a long lineage. Robert J. Moore and Raphael Arar’s book, Conversational UX Design: A Practitioner’s Guide to the Natural Conversation Framework, sets out patterns for managing talk at the sequence level and the conversation level. Openings, closings, repairs and the other moves people use to keep a conversation on track, whatever it’s about. Primary conversation handlers apply that thinking to voice AI agents in production and deal with the requirements of managing the messiness of dialogue needed in every call.
These behaviours are things you’d set at the agent-level. System prompt level expectations. They leverage the building blocks of conversation design, adjacency pairs and expandable sequences, and are required throughout a conversation to support users through their task.
The 10 key handlers and how to define them
For each handler, define what triggers it, how the agent responds and where the conversation goes next.
1. Go back/restate
People give the wrong date, pick the wrong option or answer too quickly without thinking. You’ll have to be prepared for callers to correct themselves, undo what they’ve just done or rewind the conversation to an earlier state.
2. What do you mean?
Users might not explicitly ask for “help” verbatim (though some do), but they will ask for help indirectly. “I don’t understand” or “what do you mean?” show the caller is stuck, mishear or heard you and didn’t follow. You’ll need to be prepared to re-explain, add more details or resolve contextual trouble.
3. Say that again
Voice calls are victims of two main limitations: time and attention. Callers miss things. Phone lines are noisy, kids interrupt and long numbers are hard to hold in your head. You’ll often hear “say that again” or “can you repeat that”.
4. Are you still there?
This typically happens after a period of latency. If you take too long to respond, you’ll hear “Hello?” or “Are you still there?”. The key here is to acknowledge and continue from where you left off.
5. How are you?
It’s not only “How are you?” but general manners and pleasantries. Even though many callers will know they’re speaking to an AI agent, the medium of conversation has them reciprocate small talk. Avoiding this contributes to the clunky feeling of old school chatbots.
6. What can I do?
With audio interfaces, callers can’t see a menu or a visual representation of the systems capability or progress. An enquiry into capability is common, even if the exact phrasing varies. Without an answer, they have no idea what’s in scope.
7. Hold on
Again, because of the real time nature of a voice call, you need to deal with things that emerge that prevent the flow of conversation. Users being distracted, multi-tasking and having to do something that involves their hands, ears or eyes. Callers need time to find a card or let the dog out. Intentional pauses are part of the conversation.
8. Silence
User-initiated silence is usually someone pausing to think, struggling to respond or their attention is elsewhere. Here, you have to re-engage the conversation and make sure they’re back, present and ready.
9. Speak to a human
Some users will ask to speak to a human straight away. Others will give the AI agent a go and ask for a human when they hit a roadblock. Either way, you need to handle these requests by either doing as the user asks or, even better encouraging the user to share the specifics of what they’d like to speak to an agent about. This way, you have a chance to see whether your AI agent can help them and encourage adoption.
10. Are you a robot?
Even though the EU AI Act mandates that users are informed that they’re speaking with an AI, many simply don’t hear it. We’ve had callers that go 2 minutes deep into a conversation before they get an inkling that they’re talking to a robot. However, you’ll still get this question or a variation of it. Often, it’s the caller trying to decide whether they’re going to trust you or not. So treat it as an opportunity to be transparent but also build confidence.
Where the platform sets the limits
For some handlers, the AI platform you choose can be the biggest limitation.
Take “hold on” as an example. You can write the perfect acknowledgement, but how long the agent actually waits is dictated by the platform’s capability. If that timeout is a few seconds, the caller who went to find their account number comes back to “Sorry, I didn’t catch that.” Some platforms let you extend it for a single turn. Others set it once for the whole agent.
Silence and hello (including “are you still there?”) share the same dependency. So does barge-in, which decides whether a caller can interrupt while the agent is still talking.
Platforms also ship defaults for some of these behaviours, and they’re often not what your brand would choose. If you don’t define the behaviour, you inherit someone else’s.
So check the platform settings early, before you design something it can’t deliver.
Natural conversations are built from the small stuff
These handlers are a big part of what makes a conversation feel natural. They’re among the unwritten rules of conversation that people follow without thinking and notice instantly when they’re broken.
Speech-to-speech models don’t remove the need to define them. They’re better at hearing when someone has finished speaking and at handling interruptions. But they still need to know how long to wait after “hold on”, what to say when they check-in, when to hand over to a person and how to answer “are you a robot?”. Those are design decisions, written into the prompt or the global layer.
The same applies on a cascaded system, where speech recognition, a language model and text-to-speech run as separate steps. You still define every handler, though a cascade often gives you more control over pieces such as per-turn timeouts and barge-in rules.
Whichever architecture you use, define your primary conversation handlers before you design your first journey. Write them once, configure the platform to support them and let every journey inherit them.