QA for AI-handled conversations means reviewing the tickets your AI agent closed on its own, which is the exact population your existing QA process is built to exclude. Because a bot that fails confidently closes the ticket rather than escalating it, its worst conversations leave no escalation, no human note and usually no survey response, so they never reach a review queue. Finding them means deliberately sampling the silent population first, scoring it against criteria written for a machine rather than a person, and grading it with something that did not build the bot.
In short
- A bot that fails confidently closes the ticket, so its worst conversations never escalate and never reach your review queue. The failures are invisible by construction, not by accident.
- Do not start with a random sample. Start with the tickets the bot closed alone, then the chats the customer abandoned, then the ones that came back within a week.
- An AI scorecard is not the human one with the word agent swapped in. Tone and greeting adherence stop discriminating; factual accuracy, scope adherence and failure to escalate start deciding everything.
- Deflection rate measures the absence of a human, not the presence of a resolution. A customer who gave up in frustration is counted as a success.
- When the bot is wrong, nobody owns it by default. Separate who finds the problem from who fixes it, and never put a bot’s error on a human agent’s score.
- The platform that sold you the AI agent should not be the platform grading it, because the criteria most likely to be lenient are the ones that would make the product look bad.
Why does nobody notice when an AI agent gets it wrong?
An AI agent that fails confidently closes the ticket. That one sentence explains most of why this problem stays hidden. A human who does not know the answer escalates it, asks a colleague, or leaves a note on the ticket. A model that does not know the answer produces a fluent, plausible, wrong response and marks the conversation resolved. Nothing about that outcome looks like a failure in any queue you review today.
Your review process makes it worse rather than better, and it does so by design. Most QA samples are drawn from one of three pools: conversations a human touched, tickets that were escalated, or conversations where the customer left a survey response. Every one of those filters excludes the exact population you need to see. Tickets the bot closed alone, with no escalation, no human note and no survey, are at once the largest share of your AI volume and the smallest share of anything anyone reads.
The pattern shows up in support communities almost verbatim. Teams ask for a workflow to review the tickets their AI agent handled with no human involvement, and discover the tooling quietly assumes a human was present. The stated worry is rarely the average conversation. It is the high-value customer who got an unconvincing answer, abandoned the chat rather than argue with a bot, and left nothing behind that would generate a follow-up ticket or a note for anyone to find. A second pattern recurs just as often: a team moves roughly half its ticket volume to an AI agent, the dashboard improves immediately, and churn rises a quarter later.
The expensive misses are the ones that look confident and do not escalate. This is now a named risk class rather than folklore. The NIST Generative AI Profile lists confabulation, the production of confidently stated content that is erroneous or false, as a distinct risk of generative systems, separate from the ordinary error rates you would plan for in a rules-based flow. If you have put an AI agent in front of customers, that risk class is now sitting in your queue. Read the taxonomy of silent failures before you design anything, because you cannot sample for failures you have not named.
What should you sample first?
Do not start with a random sample. Random sampling is the right instinct for a human team, where errors scatter across people, shifts and moods. AI failures are systematic. A gap in the knowledge base produces the same wrong answer hundreds of times, and a small random sample either misses it or surfaces one instance you dismiss as an outlier.
Start instead with the populations most likely to be hiding something, in this order. Each pass is a few dozen conversations, and most teams find their first real problem inside the first two.
| Pass | What you pull | Why it sits here in the order | What it exposes |
|---|---|---|---|
| 1. Closed with no human touch | Every conversation the bot resolved alone where the customer never replied again and never rated it | It is the blind spot your review queue is built to exclude, and usually your largest volume | Confident wrong answers, invented policy, silent acceptance |
| 2. Abandoned mid-conversation | Sessions where the customer stopped responding before any resolution signal | Your containment number is currently counting these as wins | Dead ends, loops, the customer who gave up rather than argue |
| 3. Came back within seven days | The bot closed it, then the same customer contacted you again on any channel | The cheapest available proxy for a wrong answer nobody flagged | Incomplete resolutions, answers that were wrong but plausible |
| 4. Escalated after three or more bot turns | Conversations that reached a human, but late | Tests whether the handoff fires at the right moment, not merely whether it fires | Late escalation, context lost in the transfer, a frustrated customer inherited by a human |
| 5. Your named and highest-value accounts | Every AI-handled conversation for the customers you cannot afford to lose | Risk is not evenly distributed, and an average score hides concentrated exposure | The commercial cost of an error, as opposed to its frequency |
| 6. Money, policy and safety | Refunds, cancellations, billing disputes, anything legal, medical or safeguarding related | These never had an acceptable failure rate, so they are checked regardless of what the first five showed | Unsafe non-escalation, policy the bot made up, commitments it had no authority to make |
What does a bot scorecard need that a human one does not?
Your human scorecard encodes assumptions about how people fail, and a model breaks those assumptions in both directions. A person may be curt under pressure but rarely invents a refund policy from nothing. A model is almost never curt and will state a fabricated policy with total composure. Point the human card at a bot and you spend your review effort on the one thing it does reliably.
The delta is smaller than people expect and sharper than they expect. Roughly six criteria change.
| On the human scorecard | What happens when you point it at a bot | What to score instead |
|---|---|---|
| Tone, warmth, empathy | Scores the prompt rather than the interaction, and barely varies between conversations | Appropriateness: did the response match the customer’s actual situation and state |
| Greeting and closing adherence | Near perfect by construction, so it inflates every score and discriminates nothing | Drop it or make it pass or fail, and stop giving it weight |
| Product knowledge | Assumes the agent either knows the answer or knows that they do not | Factual accuracy of every claim made, checked line by line against the source |
| Followed the process | A person deviates knowingly; a model drifts without noticing it has | Scope adherence: did it stay inside what it is permitted to say, do and promise |
| Escalated appropriately | A person recognises being stuck; a model can loop or dead-end indefinitely | Escalation behaviour including failure to escalate, which is the expensive half |
| Handle time | Measures agent effort, which is not a thing the bot has | Customer turns to resolution, and whether the customer reached one at all |
| No human equivalent | People hedge when they are unsure; models assert with identical confidence either way | Honesty about uncertainty: did it say it did not know instead of guessing |
How much of it do you have to score?
Sampling logic was built for a cost problem that no longer applies to this population. Teams read three percent of conversations because reading them was expensive and human error was randomly distributed, so a sample generalised. Neither half of that holds for an AI agent. Reading is now cheap enough to automate, and the errors repeat, which means the sample is not a smaller version of the truth. It is a lottery ticket on whether you happen to catch the repeating one.
Score every AI-handled conversation. That is what turns the scorecard into a monitoring instrument rather than a spot check, and it is what makes a failure that occurs in one conversation per thousand visible before it becomes a pattern in your churn data. Full coverage reveals trends that a three percent sample never could, which is the whole argument for automating the grading rather than hiring more reviewers.
Coverage is no longer a differentiator. Every vendor in this category now claims it, so treat it as the price of entry. The harder question, taken up in the last section, is whether the scores are right. Meanwhile the composition question, what actually belongs on an AI agent scorecard, and the wider discipline of agentic QA are both worth reading before you finalise criteria.
What happens at the handoff?
The handoff is where your two QA systems meet, and it is usually the one place neither of them is looking. Bot QA stops when the conversation leaves the bot. Human QA starts when the human replies. The seam between them belongs to nobody, and it is where a recoverable problem turns into a complaint.
Score three things at the seam, and score them as properties of the transfer rather than of either participant.
- Trigger timing. Did the handoff fire at the right turn? Firing too late is the common failure and the one customers remember. Firing too early is a cost problem, not a quality one, and teams routinely confuse the two.
- Context transfer. Did the human inherit what the customer already said, or did the customer have to repeat themselves? Asking a customer to restate a problem they have already explained twice is the single most reliable way to convert mild irritation into an escalation.
- What the human inherited. Did the bot make a commitment the human now has to honour or withdraw? A promised refund the human has to walk back is a worse outcome than no answer at all, and it lands on the human’s score unless you have decided in advance that it should not.
There is a second direction that gets even less attention, which is the human-to-AI handoff: a conversation a person started and a bot picked up, on a follow-up or out of hours. Scoring the handoff in both directions covers the criteria, and escalation handling quality covers the human side of the seam.
Who is accountable when the bot is wrong?
When the bot is wrong, the honest answer to who owns this is usually nobody, and that is a design flaw rather than an oversight. Three groups have a claim and each has a reason to decline it. The team that deployed the agent, often product or ops, is also the team reporting on whether the deployment worked. The support organisation inherits the fallout but did not choose the tool and cannot change the prompt. The vendor’s commercial incentive points at containment, not at candour.
The fix is not a new committee. It is separating three things that get bundled together and then dropped.
- Who owns the finding. QA does. QA’s job is to produce the evidence that a specific failure mode exists, at a measurable rate, with transcripts attached. Nothing else in this list can happen without that.
- Who owns the fix. Whoever controls the prompt, the knowledge base or the routing. Naming them in advance is what stops a finding from circulating as a screenshot for three weeks.
- Who owns the kill switch. One named person who can turn a flow off. If nobody has this, the only available response to a serious failure is a meeting.
Two rules make this durable. First, a bot’s error never lands on a human agent’s score. Scoring a person for a machine’s mistake destroys trust in the whole programme faster than anything else you could do, and once lost it does not come back cheaply. Second, every finding has to carry its evidence, because the team that owns the bot will push back on the first uncomfortable number and they are entitled to. Score traceability, showing exactly which lines in which transcript produced the deduction, is what converts an argument into a work item.
The NIST AI Risk Management Framework separates govern, map, measure and manage into distinct functions for exactly this reason: organisations that collapse them end up measuring with the same hands that manage, then wonder why nothing surfaces.
Why is deflection rate the number most likely to be lying to you?
Deflection measures the absence of a human, not the presence of a resolution. Those two things are correlated enough to look like the same measurement on a dashboard and different enough to hide a serious problem for a full quarter. Every failure mode in this article increases deflection.
Work through what actually gets counted as deflected:
- The customer who gave up. They abandoned the chat because arguing with the bot was not worth it. No human touched the ticket, so it counts as a win.
- The customer who accepted a wrong answer. They believed the bot, acted on it, and did not come back. This scores as a clean deflection and can be the most expensive conversation of the month.
- The customer who came back under a new ticket. Two contacts, one of them deflected. Deflection goes up while your real contact volume also goes up.
- The customer who went somewhere else. The strongest possible deflection number and the worst possible outcome, and nothing in the metric can tell them apart.
A deflection rate you have not sampled is a claim, not a measurement. Make it honest by pairing it with three things: a resolution signal that is independent of the bot’s own opinion, a repeat-contact rate over the following seven days, and a quality score computed on the deflected population itself rather than on the tickets that escaped it. Containment rate, measured without fooling yourself goes through the arithmetic and the denominators that quietly get excluded.
Who should grade the bot?
The uncomfortable structural question is who grades the grader. If the platform scoring your AI agent is the same platform that sold you the AI agent, its scorecard is grading its own product. The criteria most likely to end up lenient are exactly the ones that would make the product look bad: fabricated facts, unsafe non-escalation, and containment that was really abandonment. This is not bad faith, it is what happens to any measurement system pointed at its own output.
Whatever you use, three properties are not negotiable.
The grader is independent of the thing it grades
Kaizo does not build or sell AI agents. It scores the conversations they produce, human-handled and AI-handled, in the same programme and against criteria you control, which is what makes something like factual accuracy or failure to escalate scoreable without a conflict of interest. Native to Zendesk and Salesforce Service Cloud, so the conversations are graded where they already live.
The grader is validated, not trusted
An automated grader is itself a model making judgements, and it inherits the same tendency to be confidently wrong. Before you act on a single automated score, understand what an LLM-as-a-judge actually does, look at how accurate AI QA scoring really is, and learn the shape of false positives in automated QA, which is the failure that quietly creates a second review queue nobody budgeted for. The safe rollout is shadow scoring: run the automated grader in parallel with your human reviewers, compare, and only switch over when the disagreement rate is one you can explain.
Every score points at its evidence
A factual-accuracy or scope-adherence score is worth nothing if you cannot show the exact lines that failed it. Without that you cannot fix the prompt and you cannot defend the finding to the team that owns the bot. Every Kaizo score points back at the transcript that produced it, which is the difference between a number and an argument. The automated QA layer is where that evidence trail is generated, and the full method for running QA on AI agents and chatbots covers turning these passes into a standing programme rather than a one-off audit. For the metric layer on top, start with how to measure AI agent performance.
Frequently asked questions
How do you QA conversations an AI agent handled with no human involvement?
You have to pull them deliberately, because no standard queue contains them. Filter for conversations the bot closed where no agent replied, no escalation fired and no survey came back, then review them against criteria written for a machine. Start with the ones the customer abandoned mid-chat and the ones where the same customer contacted you again within seven days, since both are cheap proxies for an answer that was wrong but never flagged.
What should an AI agent QA scorecard include?
Keep whether the issue was actually resolved, whether the information was correct and whether the customer was treated appropriately. Drop tone, greeting adherence and handle time, which either do not vary or measure human effort. Add factual accuracy of every claim, policy fidelity, scope adherence, escalation behaviour including failure to escalate, and honesty about uncertainty. Those additions are the failures a person rarely produces and a model produces routinely.
Is deflection rate a good measure of AI agent quality?
No, on its own it is one of the most misleading numbers available. It records that no human touched the conversation, which is also true when the customer gave up, accepted a wrong answer, or left for a competitor. Pair it with a resolution signal that does not come from the bot, a repeat-contact rate over the following week, and a quality score calculated on the deflected conversations themselves.
Who is responsible when an AI agent gives a customer the wrong answer?
Decide it before it happens, because by default nobody owns it. Split the finding from the fix: QA produces the evidence with transcripts attached, whoever controls the prompt, knowledge base or routing owns the correction, and one named person holds the authority to switch a flow off. The one rule with no exceptions is that a bot’s error never lands on a human agent’s quality score.
How many AI-handled conversations should you review?
All of them, once the grading is automated. Sampling existed because reading conversations was expensive and human error was randomly distributed, so a sample generalised. AI failures repeat systematically, so a small sample either misses the repeating fault entirely or shows one instance that looks like an outlier. Full coverage is what makes a rare but serious failure visible before it becomes a pattern in churn.
Should the vendor that built your AI agent also grade it?
It is worth avoiding. A platform scoring its own AI agent has an interest in the outcome, and the criteria most likely to be soft are the ones that would make the product look worse: fabricated answers, unsafe non-escalation, and containment that was really abandonment. An independent grader can score those honestly, provided it shows the transcript evidence behind each deduction so the team that owns the bot can act on it.
Related terms
Find out what your AI agent closed last month
Bring a month of conversations your bot resolved with no human touch and we will run the six passes on them with you: what it got confidently wrong, where it should have escalated and did not, and how much of your deflection number is really abandonment. Every finding comes with the transcript lines behind it, so the team that owns the bot has something to act on rather than argue with.
