Build a scorecard for a good AI-handled conversation and score every one automatically. Hold it to the same scorecard as your people, and link each score to the transcript so you can fix the failure.
In short
- AI volume is too high to sample by hand.
- Look for wrong information, missed escalations, hallucinations and off-tone replies.
- Feed findings into prompts, guardrails and routing.
- Re-score afterwards to confirm the fix held.
Step 1: Define what good looks like for an AI agent
AI agents fail differently from people, so a human scorecard ported straight across will miss the things that matter most. Start by writing down what a good AI-handled conversation actually looks like for your business, in criteria a machine can judge against a transcript.
Build criteria around AI failure modes
A strong AI scorecard covers the behaviours that break trust when an agent gets them wrong:
- Factual accuracy: did the agent give correct information, or invent a policy, price, or step that does not exist?
- Resolution: did it actually solve the customer’s problem, or just close the ticket?
- Correct escalation: did it hand off to a human at the right moment, and not too late?
- Tone and empathy: did it match the customer’s emotional state instead of staying robotic?
- Policy and safety adherence: did it stay inside the boundaries you set, refusing what it should refuse?
Make each criterion evidence-checkable
Write criteria so that a pass or fail can be traced to a specific line in the conversation. Kaizo lets you build the scorecard your business actually uses and judges each conversation against it, the same way your best reviewer would, which keeps AI QA and human QA on one comparable standard.
Step 2: Score 100% of AI conversations, not a sample
Manual QA samples 2% to 5% of tickets because a person can only read so many. AI agents make that ceiling untenable: they run at volumes no review team can sample meaningfully, and a rare but damaging failure will almost always live in the 95% nobody reads.
Automate the scoring
Connect the helpdesk or CRM where your conversations already live, then let the system score each AI-handled interaction as it lands, continuously and in the background. Kaizo integrates natively with Zendesk and Salesforce and reads conversations straight from the systems you already run, so there is no separate pipeline to maintain. Our page on QA for AI agents shows how Kaizo scores AI agent conversations on the same scorecard as your team’s.
Why full coverage matters more for AI
A single agent applies the same behaviour across thousands of conversations, so one systematic flaw, a misread policy or a bad default, repeats at scale. Scoring 100% is what surfaces that pattern early. At UiPath, Kaizo automated 100% of QA with 200% ROI, which is the level of coverage AI-handled volume now demands.
Step 3: Make every score checkable
This is the step most teams miss, and it is the one that decides whether your AI QA can be trusted. A score nobody can verify will not survive the first disagreement, whether it comes from the team that owns the bot or from leadership. Every score has to be traceable to the conversation and open to challenge.
Evidence is what makes the score credible
The evaluator should apply the same scorecard whether a conversation was handled by a human, your own bot, or a third party’s AI. Kaizo scores 100% of human and AI-agent conversations against your own scorecard, and every score links to the evidence in the conversation, so its verdict on an agent can be checked rather than taken on faith.
What to ask a QA vendor
- Can you score an AI agent built by someone else on the same scorecard as ours?
- Does every score link to the evidence, so we can check your judgment rather than take it on faith?
Step 4: Catch the AI-specific failure modes
Once scoring runs on every conversation, point it at the failures that are unique to, or amplified by, automation. These are the categories worth isolating and tracking over time.
| Failure mode | What it looks like | What the scorecard checks |
|---|---|---|
| Wrong information | The agent states a policy, price, or step that is false | Was every factual claim correct and grounded in your sources? |
| Missed escalation | The agent keeps handling a case it should have handed to a human | Did it escalate at the right trigger and in time? |
| Hallucinated confidence | A wrong answer delivered as if it were certain | Did the agent avoid inventing details it could not verify? |
| Off-tone reply | Robotic or dismissive language during a frustrated contact | Did tone match the customer’s emotional state? |
| Policy breach | The agent does something it was told to refuse | Did it stay inside its safety and policy boundaries? |
Step 5: Feed findings back and re-score
QA is only worth doing if it changes behaviour. Because every Kaizo score links to the exact moment in the transcript that produced it, a failure is not just a flag, it is a specific, reproducible example you can act on.
Close the loop
- Route accuracy failures back to the prompt, knowledge base, or grounding source that produced the wrong answer.
- Turn missed escalations into a tightened routing rule or trigger.
- Group recurring failures so you fix the pattern once instead of patching cases one by one.
Then re-score to confirm the fix held. Full, continuous coverage means you see the effect of a change across all conversations, not in a sample that might miss the regression.
Common mistakes to avoid
A few errors quietly undermine AI QA programs before they deliver value.
- Sampling AI the way you sampled humans: a 2% sample of a high-volume bot misses systematic failures almost by design.
- Holding AI to a softer standard than people: if the bot is scored on looser criteria, a good-looking score is not the same as good work.
- Scoring without evidence: a number you cannot trace to a line in the transcript cannot be verified, coached on, or challenged.
- Reusing the human scorecard unchanged: AI failure modes like hallucination and false confidence need their own criteria.
- Measuring but never feeding back: QA that does not change prompts, guardrails, or routing is just reporting.
Frequently asked questions
How do you QA an AI agent or chatbot?
Build a scorecard that defines a good AI-handled conversation, covering accuracy, resolution, correct escalation, tone, and policy adherence. Score 100% of AI conversations against it automatically, hold it to the same standard as your human agents, and link every score to the evidence in the transcript so failures can be verified and fixed.
How do you know an AI agent’s QA score is accurate?
Check it. An evaluator that links every score to the evidence in the conversation is what makes the score credible, because anyone can verify it, and comparing automated scores with your own reviewers’ scores on the same conversations shows how often they agree.
Can you use the same scorecard for AI agents and human agents?
You can share the criteria that apply to both, like resolution and tone, which keeps AI and human QA comparable. But AI needs extra criteria for its own failure modes, such as hallucination and false confidence, so the strongest approach is one standard with AI-specific checks added.
How much of AI conversations should you QA?
All of it. AI runs at volumes no team can sample meaningfully, and a systematic flaw repeats across every conversation the agent handles, so a small sample will usually miss it. Automated scoring of 100% of conversations is what surfaces those patterns early.
