Framework: Choosing a voice for your voice AI agent

Most teams choose a voice for their voice AI agent the same way. Someone opens a voice gallery, plays a handful of samples and picks the one that “sounds about right.” Or worse, the highest paid person has a personal preference and you’re told to just run with that.

With a voice AI agent, the sound of the voice is all you have. It’s literally the interface. It’s all the user hears and it’s the only thing you have to create a great experience. You should therefore select that voice based on more than vibes or the manager’s preference.

Here is the framework we use to replace the feeling with a process. It will not tell you which voice to pick. But it will ask you a series of questions about the qualities you want and, crucially, why you’re picking it. This is helpful not only in creating the best experience for your users, but also in enabling all of your stakeholders to get behind the process and support your efforts.

Why the voice of your AI agent carries more weight than teams assume

In a chat interface, you have a full toolkit for making something easy to use. Buttons, suggestion chips, embedded video, hyperlinks, the ability to read the page a user is on and contextualise the conversation around it. A chatbot has a dozen affordances working alongside the words on screen.

A voice AI agent has one: the voice.

That is the whole interface. There is no scroll back, so a caller can’t glance up to remember what was just said. There is no URL, so there is no ambient sense of where you are in a process. Everything a chat interface leverages a screen for, a voice interface has to carry in tone, pacing and clarity alone.

Callers are rarely confused because voice conversations are inherently messier than chat ones, contrary to popular opinion. Users are constantly trying to find their bearings and orient themselves, and they only have audio with which to do it. So your voice is the only tool available to help them.

That voice has to do several jobs at once. It has to:

1. Signal that this is a genuine interaction with the company, not a low-effort automation or spam.
2. Show that it understands the caller and can actually help them.
3. Build trust with the user so that they persevere with the interaction rather than escalating to a human agent.
4. Flex with whatever the conversation demands, from delivering bad news or drawing out more detail from someone who has not said enough for the system to help them yet.

Pick a voice without accounting for any of that and you are gambling the entire user experience on vibes.

The Baseline: pass or fail

Most people think that a voice AI agent just has to sound human-like, and that’s all there is to it. However, there are three factors that matter much more than sounding human. You shouldn’t even consider a voice that doesn’t meet these three criteria. I’d rather go with a robotic voice that passes these three, than a human-like voice that doesn’t.

1. Clarity. The voice has to be understood, consistently, including when it says your brand name or your product names. This sounds obvious until you actually test it. Some voice models have quirks baked into the base training that no amount of tuning gets out. It is a bit like recording a band and having a plane fly overhead mid-take. The noise is baked into the signal and you are better off re-recording than trying to scrub it out afterwards. If a voice mangles your product name in the demo, that is not a small fix. You can resolve some of this with SSML and markup (different providers have different ways of doing this today), but you need to test this first to make sure that the things you need to say are able to be said in a way that’s understood clearly.
2. Technical performance. A voice that takes several seconds to generate a response isn’t going to work, no matter how good it sounds. Aim for a response latency around 300 milliseconds as your baseline. And test it under phone conditions, not headphones. Audio on a phone call is compressed down to roughly 8kHz, a fraction of the quality you hear through a browser or a headset, so a voice that sounds crisp in a demo can sound noticeably different once it is squeezed through an actual call.
3. Accessibility. The voice needs to meet accessibility standards, including speaking rate. This is not optional polish. It is part of the baseline. Yet, many providers don’t provide this out of the box, unless you’re using a speech to speech model such as GPT-Live-1. This has capability baked in, but many don’t.

Score these as either a pass or fail. There is no partial credit for a voice that sounds trustworthy but cannot pronounce your own product name.

The Character Assessment

Once a voice clears the baseline, the harder and more subjective work starts. This is where “sounds right” needs to be pulled apart into its component parts and each part assessed. The challenge of this part is that it can be subjective, so it’s always best to have three people do this assessment. That way, you can compare and contrast and find consensus.

Anything below a three is a no-go. And a three still needs work. Above three, you can live with it.

1. Brand alignment. Does the voice sound like your brand? If your chatbot already has a defined character, the voice needs to contribute to that same character rather than introduce a different one down the phone. If it doesn’t, you need to consider what your brand should sound like based on your brand guidelines, your customer base and your use case. Ideally, before this exercise, you’ve already documented your intended AI agent personality.
2. Credibility. Does it sound trustworthy and authoritative? A voice can be perfectly clear and still not sound like someone whose advice you would follow. Picture a happy, jovial entertainment-based voice answering your support line. It might be understandable and warm, but you still wouldn’t trust it to talk you through your accounts.
3. Tone. This is the emotional register of the voice and it should match what the caller is actually going through. Someone ringing with a problem does not want an overenthusiastic voice on the other end. Some providers offer voices trained for gaming or film characters, built for drama and spectacle. That tone doesn’t belong on a support call, however polished it sounds in isolation.
4. Register. How formal or casual is the voice? A voice can hit the right tone and still feel too stiff, or too casual, once you compare it against how the brand is meant to sound everywhere else and when you compare it to the use case and context it’s used within.
5. Naturalness. This is really tone and prosody combined, but here I’m talking more about prosody. Rhythm, speed and inflection of speech. Too fast and callers can’t follow it. Too slow and they disengage. Some accents naturally rise at the end of a sentence in a way that can make a statement sound like an unintended question. Whether that fits depends entirely on what you are asking the voice to say.
6. Locale and accent. A voice trained on one regional variant of a language does not automatically transfer to another. A French Canadian voice does not sound native to a caller in France. Within a single language you can find flavours that lean toward one region over another and one might fit your audience or use case better.
7. Demographic fit. Does the implied age and life stage of the voice match what it is being asked to do? And does it match the expectations of the user based on the use case? A young voice talking someone through their accounting software is an obvious mismatch once you say it out loud.

Together, these explain what “sounds right” means. When you score them individually, you’re able to properly discuss the qualities of the voice you care most about. It’s not black and white, but it provides the space for you to discuss and debate internally to arrive at a voice that’s been chosen for a specific purpose, based on specific criteria, that you all agree on.

Running the process

You cannot run a full scorecard against every voice out there. Eleven Labs alone has thousands.

1. Start with intuition. Get two or three people to go through the voice gallery from a variety of providers. Have them each shortlist the ones that feel about right. This first pass isn’t rocket science, but it does take a bit of effort. You will immediately rule out voices that are obviously wrong. The narrator built for audiobooks, the overly theatrical character, the one that mispronounces your product name in the first three seconds.
2. Run The Baseline on your long list. A first cut might take you from dozens of options down to ten. A second pass, testing a bit more deliberately by running through the baseline assessment above will bring that down further.
3. Run The Character Assessment. By the time you are down to a final three-to-five, you run the full Character Assessment scorecard against each one. If three people do this independently, you’ll find the consensus when you bring it back together.

Also, test them against the situations that will actually occur in conversation: an objection, a frustrated caller, a piece of bad news the bot has to deliver without sounding careless about it. A voice that sounds warm reading a scripted greeting can sound completely wrong once it has to say no to someone.

Bring your final shortlist to the stakeholders who need to sign off, and make the decision with the scorecard in front of you.

The point of the framework

The value of this process is not that it guarantees the best possible voice. It is that it replaces “it sounded right” with a set of criteria everyone can see, debate and agree to.

When someone in the business questions the choice six months later, you have an answer grounded in clarity, performance, accessibility, brand fit, credibility, tone, register, naturalness, locale and demographic fit, rather than a vibe.

Voice is the entire interface in a voice AI agent. Treat the selection with the rigour it deserves.