To QA AI agents and chatbots, build a scorecard that defines a good AI-handled conversation, then score every one of those conversations against it automatically rather than sampling a handful by hand. The grader must be independent from the AI it is grading, so no vendor is marking its own homework, and every score must link back to the evidence in the transcript so you can verify it and fix the failure.
In short
- Define a scorecard for AI-handled conversations: accuracy, resolution, correct escalation, tone, and policy adherence.
- Score 100% of AI conversations automatically, because AI volume is far too high to sample by hand.
- Keep the grader neutral: the AI that evaluates a conversation must be independent from the AI that handled it.
- Hunt the specific AI failure modes: wrong information, missed escalation, hallucination, and off-tone replies.
- Link every score to the exact moment in the transcript, so a failure can be verified and fixed, not just flagged.
- Feed findings back into prompts, guardrails, and routing, then re-score to confirm the fix held.
Step 1: Define what good looks like for an AI agent
AI agents fail differently from people, so a human scorecard ported straight across will miss the things that matter most. Start by writing down what a good AI-handled conversation actually looks like for your business, in criteria a machine can judge against a transcript.
Build criteria around AI failure modes
A strong AI scorecard covers the behaviours that break trust when an agent gets them wrong:
- Factual accuracy: did the agent give correct information, or invent a policy, price, or step that does not exist?
- Resolution: did it actually solve the customer’s problem, or just close the ticket?
- Correct escalation: did it hand off to a human at the right moment, and not too late?
- Tone and empathy: did it match the customer’s emotional state instead of staying robotic?
- Policy and safety adherence: did it stay inside the boundaries you set, refusing what it should refuse?
Make each criterion evidence-checkable
Write criteria so that a pass or fail can be traced to a specific line in the conversation. Kaizo lets you build the scorecard your business actually uses and judges each conversation against it, the same way your best reviewer would, which keeps AI QA and human QA on one comparable standard.
Step 2: Score 100% of AI conversations, not a sample
Manual QA samples 2% to 5% of tickets because a person can only read so many. AI agents make that ceiling untenable: they run at volumes no review team can sample meaningfully, and a rare but damaging failure will almost always live in the 95% nobody reads.
Automate the scoring
Connect the helpdesk or CRM where your conversations already live, then let the system score each AI-handled interaction as it lands, continuously and in the background. Kaizo integrates natively with Zendesk and Salesforce and reads conversations straight from the systems you already run, so there is no separate pipeline to maintain.
Why full coverage matters more for AI
A single agent applies the same behaviour across thousands of conversations, so one systematic flaw, a misread policy or a bad default, repeats at scale. Scoring 100% is what surfaces that pattern early. At UiPath, Kaizo automated 100% of QA with 200% ROI, which is the level of coverage AI-handled volume now demands.
Step 3: Keep the grader neutral (do not mark your own homework)
This is the step most teams miss, and it is the one that decides whether your AI QA can be trusted. If the same vendor both sells you the AI agent and grades that agent’s work, it has a built-in conflict of interest. A system that profits from the agent looking good is not a neutral judge of whether it is good.
Independence is what makes the score credible
Effective QA of AI agents has to be neutral by design. The evaluator should have nothing to protect when it scores a conversation, whether that conversation was handled by a human, your own bot, or a third party’s AI. Kaizo does not sell AI agents, so it can grade them without conflict, which is precisely why its verdict on an agent means something.
What to ask a QA vendor
- Do you also sell the AI agents you would be grading?
- Can you score an AI agent built by someone else on the same scorecard as ours?
- Does every score link to the evidence, so we can check your judgment rather than take it on faith?
Step 4: Catch the AI-specific failure modes
Once scoring runs on every conversation, point it at the failures that are unique to, or amplified by, automation. These are the categories worth isolating and tracking over time.
| Failure mode | What it looks like | What the scorecard checks |
|---|---|---|
| Wrong information | The agent states a policy, price, or step that is false | Was every factual claim correct and grounded in your sources? |
| Missed escalation | The agent keeps handling a case it should have handed to a human | Did it escalate at the right trigger and in time? |
| Hallucinated confidence | A wrong answer delivered as if it were certain | Did the agent avoid inventing details it could not verify? |
| Off-tone reply | Robotic or dismissive language during a frustrated contact | Did tone match the customer’s emotional state? |
| Policy breach | The agent does something it was told to refuse | Did it stay inside its safety and policy boundaries? |
Step 5: Feed findings back and re-score
QA is only worth doing if it changes behaviour. Because every Kaizo score links to the exact moment in the transcript that produced it, a failure is not just a flag, it is a specific, reproducible example you can act on.
Close the loop
- Route accuracy failures back to the prompt, knowledge base, or grounding source that produced the wrong answer.
- Turn missed escalations into a tightened routing rule or trigger.
- Group recurring failures so you fix the pattern once instead of patching cases one by one.
Then re-score to confirm the fix held. Full, continuous coverage means you see the effect of a change across all conversations, not in a sample that might miss the regression.
Common mistakes to avoid
A few errors quietly undermine AI QA programs before they deliver value.
- Sampling AI the way you sampled humans: a 2% sample of a high-volume bot misses systematic failures almost by design.
- Letting the agent vendor grade itself: without an independent evaluator, a good-looking score is not the same as good work.
- Scoring without evidence: a number you cannot trace to a line in the transcript cannot be verified, coached on, or challenged.
- Reusing the human scorecard unchanged: AI failure modes like hallucination and false confidence need their own criteria.
- Measuring but never feeding back: QA that does not change prompts, guardrails, or routing is just reporting.
Frequently asked questions
How do you QA an AI agent or chatbot?
Build a scorecard that defines a good AI-handled conversation, covering accuracy, resolution, correct escalation, tone, and policy adherence. Score 100% of AI conversations against it automatically, keep the grader independent from the AI being graded, and link every score to the evidence in the transcript so failures can be verified and fixed.
Why does the grader need to be independent from the AI agent?
If the same vendor sells the AI agent and grades it, it is marking its own homework and has an incentive to make the agent look good. A neutral evaluator has nothing to protect when it scores a conversation, which is what makes the score credible. Kaizo does not sell AI agents, so it can grade them without conflict.
Can you use the same scorecard for AI agents and human agents?
You can share the criteria that apply to both, like resolution and tone, which keeps AI and human QA comparable. But AI needs extra criteria for its own failure modes, such as hallucination and false confidence, so the strongest approach is one standard with AI-specific checks added.
How much of AI conversations should you QA?
All of it. AI runs at volumes no team can sample meaningfully, and a systematic flaw repeats across every conversation the agent handles, so a small sample will usually miss it. Automated scoring of 100% of conversations is what surfaces those patterns early.
Related terms
See AI agent QA on your own conversations
Bring a week of your real AI-handled conversations and we will show you 100% coverage, the failure modes we catch, and a neutral score you can trace to the transcript.
