An AI agent scorecard is the set of criteria you use to evaluate conversations handled by an automated system rather than a person. It is not the human QA scorecard with the word agent swapped in, because an AI agent fails in ways a person does not and cannot fail in some of the ways a person can. A good AI agent scorecard drops the criteria that only make sense for a human, keeps the ones about whether the customer’s problem was actually solved, and adds a set of criteria unique to automation: hallucination, unsafe escalation behavior, scope adherence, and honest handling of the limits of what the agent knows.
In short
- An AI agent scorecard is not a human scorecard reused. Some human criteria stop applying, and new failure modes have to be added.
- Drop criteria that measure human effort or affect, such as tone warmth as a proxy for empathy, and replace them with checks on correctness and appropriateness.
- Add criteria a human rarely triggers: making up facts, inventing policy, escalating unsafely or not at all, and acting outside its remit.
- Keep the criteria that matter regardless of who is answering: was the issue resolved, was the information correct, was the customer treated well.
- Score every AI conversation rather than a sample, because automated failures repeat at scale and the worst ones are silent.
- The platform scoring the AI agent should not be the one that sold it, or the scorecard grades its own product. Kaizo does not sell AI agents, so it grades them as a neutral party and traces every score to the transcript.
Why you cannot reuse the human scorecard
The instinct when you deploy an AI agent is to score it with the scorecard you already have. It is tempting because the goal looks the same: a good customer conversation. But the scorecard you built for people encodes assumptions about how people fail, and an AI agent breaks those assumptions in both directions.
A human agent will not usually invent a refund policy that does not exist, but they may be curt under pressure. An AI agent will almost never be curt, but it will state a fabricated policy with complete confidence. Scoring the AI agent on tone warmth tells you little, because tone is the thing it does most reliably. Scoring it on whether every factual claim it made was true tells you almost everything, because that is where it actually fails. The scorecard has to move its attention to where the risk now lives. Our full method for QA on AI agents and chatbots covers the workflow around this; the question here is narrower: what actually goes on the card.
Human criteria that stop making sense
Start by removing the criteria that measure something an AI agent does not have or does not vary on.
- Tone and warmth as a proxy for empathy: an AI agent’s tone is set by its prompt and barely varies, so scoring it measures the configuration, not the interaction. Empathy still matters, but as whether the response was appropriate to the customer’s state, which is covered below.
- Effort and initiative: criteria like going the extra mile assume a person choosing to do more. They do not map onto a system executing instructions.
- Adherence to a script the human memorized: replaced by scope and policy adherence, which is a different and sharper check for a machine.
- Personal development criteria: anything about the agent learning or improving belongs to the model owner, not to a per-conversation score.
Removing these is not lowering the bar. It is pointing the bar at the failures that are actually possible.
Criteria that only apply to an AI agent
These are the additions that make it an AI agent scorecard rather than a repurposed human one. Each one describes a failure a person rarely produces and a machine produces routinely.
| Criterion | What it checks | Why a human rarely triggers it |
|---|---|---|
| Factual accuracy of every claim | Nothing the agent stated was invented or wrong | People hedge when unsure; models assert with equal confidence whether right or wrong |
| Policy fidelity | The agent did not invent, soften, or overstate a policy | A human knows they do not know a policy; a model will generate a plausible one |
| Scope adherence | The agent stayed within what it is allowed to do or promise | People sense the edge of their authority; a model has to be told and can drift past it |
| Escalation behavior | It handed off when it should have, and did not trap the customer | A human recognizes being stuck; a model can loop or dead-end without noticing |
| Honesty about uncertainty | It said it did not know rather than guessing | Guessing confidently is a model default, not a human one |
| Safe handling of sensitive requests | It refused or routed self-harm, fraud, or legal-risk cases correctly | Judgment a person applies instinctively has to be an explicit criterion for a machine |
Criteria that stay the same
Some things matter regardless of who or what is answering, and these are the spine of the scorecard.
- Was the issue resolved: the customer left with their problem solved, not merely without a human. This is the criterion a containment rate quietly skips.
- Was the information correct and complete: shared with the human scorecard, but far heavier here.
- Was the customer treated appropriately: reframed from warmth to appropriateness, whether the response matched the customer’s situation and emotional state.
- Was the interaction efficient for the customer: not agent handle time, but whether the customer got there without unnecessary loops.
Writing a rubric an AI can score covers how to phrase each of these so it is checkable against the transcript rather than a matter of opinion.
How to score against it: coverage and traceability
Two things separate a working AI agent scorecard from a decorative one.
Score everything, not a sample
Human QA sampled a few conversations per agent because reading them was expensive. That logic breaks for AI agents, because their failures are systematic: a prompt weakness that produces one hallucination produces it hundreds of times, and a 2% sample will usually miss the pattern until it is already a problem. Scoring 100% of conversations is what makes the scorecard a monitoring instrument rather than a spot check. At UiPath, Kaizo automated 100% of QA with 200% ROI and an 8% lift in quality score.
Trace every score to the transcript
A factual-accuracy or policy-fidelity score is only useful if you can see the exact lines that failed it. Otherwise you cannot fix the prompt, and you cannot defend the score when the team that owns the agent pushes back. Every Kaizo score points at the evidence that produced it.
Grade the agent with something that did not build it
The uncomfortable structural point: if the platform scoring your AI agent is the same one that sold it, its scorecard is grading its own product, and the criteria most likely to be soft are exactly the ones that would make the product look bad. Kaizo does not sell its own AI agents. It scores the conversations they produce as a neutral party, which is the whole reason a criterion like factual accuracy or unsafe escalation can be scored honestly. The silent-failure taxonomy goes deeper on the failures that only a neutral grader tends to name.
Frequently asked questions
What is an AI agent scorecard?
It is the set of criteria used to evaluate conversations handled by an automated system rather than a person. It differs from a human QA scorecard because it drops criteria that only apply to people, keeps the ones about whether the customer’s issue was resolved and the information was correct, and adds criteria unique to automation such as factual accuracy, policy fidelity, scope adherence, and escalation behavior.
Can I use my existing QA scorecard for an AI agent?
Not without changing it. A human scorecard encodes assumptions about how people fail, and AI agents fail differently. Tone warmth barely varies for a model and tells you little, while fabricated facts and invented policy, which humans rarely produce, become the main risk. Reusing the human scorecard points your attention at the wrong failures.
What criteria are unique to scoring an AI agent?
Factual accuracy of every claim, policy fidelity (not inventing or overstating a policy), scope adherence, escalation behavior (handing off when stuck rather than trapping the customer), honesty about uncertainty, and safe handling of sensitive requests. Each describes a failure a human rarely produces and a model produces routinely.
Why should the AI agent scorecard not come from the vendor that built the agent?
Because that scorecard would be grading its own product, and the criteria most likely to be soft are the ones that would make the product look bad. A neutral grader that does not sell AI agents can score factual accuracy and unsafe escalation honestly, and trace each score to the evidence in the transcript.
Related terms
Build an AI agent scorecard that catches what matters
Bring the conversations your AI agent handled last month and the scorecard you use today. We will show you which criteria stop making sense for a machine, which failures your current card cannot see, and what a scorecard built for automation catches instead. Because Kaizo does not sell its own AI agents, every score traces back to the exact lines in the transcript.
