When an AI agent gives a bad answer, the model is often blamed: it hallucinated, went off topic or even made something up. Rasmus Hauch, CTO at Boost.ai, thinks that’s usually the wrong diagnosis.
In a recent conversation on the VUX World podcast, Rasmus said that in the large majority of cases, the AI is working exactly as it should. He estimated that in around 99% of cases, the AI models are not the cause of the issues.
The rest come from the things around it: the context, the systems it connects to, the knowledge it draws on, the guardrails and the process that sits inside.
Where the failures actually come from
Rasmus was clear about the pattern he sees. This is why Rasmus talks about orchestration rather than flows. On the Boost.ai platform, a single conversation can switch between 100-200 different agents, each handling its own topic. The control plane is the layer that decides which agent runs when. Get the “plumbing” wrong and that whole layer has nothing solid to stand on.
When an agent gives a bad answer, it is usually a context, system, or knowledge problem. The model did what it was told. And it was likely told the wrong thing, pointed at the wrong data, or couldn’t reach the backend it needed.
The biggest limiting factors in conversational AI projects are often external. If your process is manual and your data is disconnected, it doesn’t matter how good the model is. You have to go back into the business, find the waste and remove it before the conversation layer can do anything useful.
“The traditional conversational tree is more or less dead, and the job is really in the loop now, or the control plane.”
Rasmus Hauch, CTO, Boost.ai
Your source data is probably contradicting itself
Rasmus gave a specific reason why this is hard. Most organisations don’t have a single version of the truth.
There’s a support article that says one thing. An internal procedure that says something slightly different. A chat log somewhere that says a third thing. When you train an agent on all materials, you are training it on a contradiction. Someone has to sit down and decide what the answer should be for each case. That is human work, and there’s a lot of it.
Teams tend to underestimate this. The model works. The material feeding it can come from multiple versions of the answer, and a person has to resolve which one is right.
Testing is an ongoing process, not a phase at the end
If the model is rarely the problem, then proving the whole system works becomes one of the key events.
Boost.ai builds personas and runs them at scale. Thousands of digital and voice conversations, with a synthetic voice acting as the customer, to test how the platform responds. Running the conversations is the easier part. The harder part is determining whether those thousands of conversations actually worked, which requires setting up clear evaluation criteria in advance.
Rasmus framed testing as something woven through the build. You write tests, build a bit of the agent, test again, build more, test again. He borrowed the idea from test-driven development in software. The point for customer experience (CX) leaders is simpler: if you can’t test it at scale before it goes live, you don’t know what you’re shipping.
And the first thing that happens in production is the customer says something you didn’t plan for. That is not a failure of the model. It’s the reason the trust loop exists, feeding real conversations back into the build so the system keeps improving with a human in the loop.
Rasmus is wary of this shortcut. Clients sometimes want to plug in their own fine-tuned or third-party speech models. He usually advises against it. Once you’re running millions of conversations, mixing providers makes it hard to keep operations stable, and when something breaks, you can’t identify a single owner. His view is that a customer should be able to come to Boost.ai, say “Fix this”, and have one team fix it.
Why this matters for regulated sectors
For banks, insurers and public sector teams, one wrong answer can make headlines, so the question of who’s responsible comes up quickly.
Rasmus was careful here and said plainly he isn’t a lawyer. His framing was around shared responsibility. The EU AI Act sets out three core roles: general-purpose model providers, AI providers and AI deployers. Rasmus puts Boost.ai’s customers in the deployer role, since they’re the ones rolling these services out to the public.
Boost.ai’s job is to give customers the tools to set things up safely and to be accountable when its own technology falls short. The business still owns the decisions its agent makes on its behalf.
He also pointed out that the platform is not a decision engine. When money moves from one account to another, it happens through a backend system with predefined steps, not because a probabilistic model decided to do so. For high-risk actions, you still want a hard yes-or-no, a button, or a rule, not a generated answer. That distinction is doing a lot of quiet work in keeping these systems safe.
What no model can be trained for
I put a vision to Rasmus: agents building agents. One agent maps the customer interaction, another determines which tools are needed, another writes the tests, another runs them, and another makes the improvements. A world where the whole build loop runs itself.
Rasmus sees real blockers. The first is the source data problem already covered: customers don’t have one clean version of the truth for an agent to learn from. The second is that no single person is good at everything, so there’s no one human whose knowledge you could simply clone into an agent. The third is that the world keeps changing in ways no model has seen.
Rasmus gave one example that stood out. A Boost.ai customer in Iceland, an airline, faced a volcanic eruption. Five thousand people were standing in the airport, not knowing what to do.
No model is trained for that. The world changes faster than the training data. Business strategy shifts, a bank moves into a new market, or an unplanned event lands overnight. Each of those needs a human to decide what the agent should say at this point. That is why Rasmus doesn’t expect the human role to disappear, even ten years out. The environment the models live in never stops moving.
The takeaway for CX leaders
If your AI project is struggling, the instinct is to look at the model. Rasmus’s experience says look everywhere else first.
Fix the source data. Connect the systems. Remove the manual steps in the process. Set up testing you can trust before go-live, and a feedback loop afterwards. Decide, deliberately, what the agent is allowed to say and do. Get those right, and the model tends to hold up. Get them wrong, and no model will save you.
Technology is one of the easiest parts now. The key work is everything else around it.
Listen to the full episode with Rasmus Hauch on the VUX World podcast.
