Think of your new AI Phone Assistant like a new intern who started working for you.
At the start you give them lots of information and rules, then watch over their shoulder for the first week or so until you're happy that they're acting as you'd expect.
Over the next few weeks you reduce the amount of time that you spend looking over their shoulder, but you never walk away completely and leave them to it.
There will always be new scenarios that will come up on calls which require your guidance.
What to look at first
1. Call reason patterns
What are the top 3 reasons people called this week?
Are those reasons covered clearly in your Knowledge Base?
Does your assistant have a simple process for each?
2. Outcome quality
For each call, ask:
Did the caller get an answer, or a clear next step?
Did the assistant capture the minimum required details?
Did the assistant stay within scope?
3. Moments of friction
These show up as:
Caller repeats the same detail
Caller asks “are you a real person?” multiple times
Caller sounds frustrated, or says “that is not what I asked”
The assistant gives a long explanation instead of acting
The assistant pauses for a long time
How to diagnose the root cause
When you see an issue, decide which bucket it fits:
Bucket A: Missing or unclear facts
Symptoms: vague answers, wrong policy details, “it depends” replies
Fix: update the Knowledge Base with a short, phone-friendly answer.
Bucket B: Process problem
Symptoms: assistant does not ask the right questions, or does not confirm next step
Fix: update Behaviours with a simple step-by-step flow.
Bucket C: Conflicting instructions
Symptoms: assistant follows one rule sometimes, then ignores it. Sometimes there is an extended silence as the assistant struggles with conflicting instructions.
Fix: remove duplicates, make one instruction the source of truth, keep wording unambiguous.
Bucket D: Model variability (non-determinism)
Symptoms: mostly correct, occasional odd phrasing, small differences in closing line
Fix: make the instruction more concrete, add an example, and test it with multiple calls.
A quick review method that works
Use this routine whenever you review a call:
Read the transcript once, without editing anything.
Highlight the first moment the call went off-track.
Decide the smallest possible change to prevent that issue next time.
Make the change, then run a test call for that exact scenario.
Examples of “small changes” that work well
Replace “be helpful and friendly” with “use short sentences, ask one question at a time, confirm the next step before ending.”
Replace “offer a callback” with “offer a callback if the caller asks for help that needs a human, or if they sound unsure, ask for preferred time and preferred contact method.”
Replace a long website paragraph with 3 bullet points and a single next step.
How many calls should you review?
First week: 10 to 20 calls per day until patterns stabilise.
After that: 10 calls per week is enough for most SMEs, as long as you act on what you find.