A QA criterion is ready for automated scoring when two reviewers, given only the transcript and the exact wording of the criterion, would independently reach the same verdict. Criteria written for humans usually fail that test, because a human reviewer silently fills the gaps with context, history and judgment, while a machine applies exactly what you wrote and nothing more. The fix is mechanical rather than technical: anchor every criterion to observable evidence in the conversation, split compound criteria into single checks, define what does not apply, and test the wording on a handful of real conversations before you roll it out. Most complaints that start with the AI got it wrong end at a criterion that two humans would have read two different ways.
In short
- A criterion is automatable when two reviewers with only the transcript and the wording would reach the same verdict. If they would not, the criterion is not ready.
- Vague criteria do not fail loudly. They fail inconsistently, which looks like an unreliable model when the real problem is the rubric.
- Anchor criteria to observable evidence in the conversation, not to inferred intent or to how the agent seemed to feel.
- One criterion, one check. Compound criteria joined by and or or cannot produce a meaningful single score.
- Not applicable is the most commonly forgotten case: a criterion that cannot apply must never silently count as a fail.
- Subjective criteria like tone and empathy are still writable, as long as you anchor them to observable behaviors rather than to feelings.
- Version your criteria. Changing the wording changes what the score means, and unversioned edits quietly break your trend data.
Why rubrics written for humans break when you automate them
Every QA program starts with criteria written for people, which is the right place to start and also why the first attempt at automated scoring so often disappoints. A human reviewer reads a criterion like handled the issue well and quietly supplies everything the sentence leaves out: what the customer was asking for, what this team counts as handled, what the agent could reasonably have done, how a similar conversation was scored last month. None of that is written down. It lives in the reviewer’s head.
A machine has none of that context unless you put it in the criterion. It applies exactly what you wrote, the same way, on every conversation. That is the point of automating, and the moment your rubric’s ambiguity stops being invisible.
Vague criteria fail inconsistently, which is worse than failing loudly
A criterion that is simply wrong gets spotted and fixed. Ambiguity does something more damaging: it produces a defensible verdict on some conversations and an indefensible one on others, with no visible pattern. Agents dispute the odd score, reviewers cannot explain the difference between two similar cases, and everyone concludes the AI is unreliable. In most cases the scoring engine did exactly what it was told, and the instruction was the unreliable part.
The distinction matters because the fix follows from it. If the model were the problem, your options are to accept the error rate or change vendors. If the wording is the problem, you can fix it this afternoon.
This article is about that craft: turning one criterion into something a machine can evaluate the same way twice. For the ground level, see what a QA rubric is and the wider QA scorecard guide, or adapt ready-made QA scorecard templates. What follows assumes you have a rubric and it is not scoring the way you expected.
The test every criterion has to pass
There is one test, and it costs nothing to run:
Could two people, given only the transcript and this exact wording, independently reach the same verdict?
Not two people who sat in the same rubric workshop and absorbed the same unwritten assumptions. Two people who have only the words in front of them. If the answer is no, the criterion is not ready to automate, and it was probably not scoring consistently under humans either.
Run the test literally rather than in your head. Hand a criterion and five real conversations to two colleagues with no other instructions, and compare verdicts. Where they disagree, do not argue about who was right. Ask what in the wording allowed two readings, and rewrite until that gap closes. It is the same discipline as a QA calibration session, applied to one criterion at a time.
A criterion that passes has four properties: it names the evidence a reader should look for, it holds one decision rather than two joined by and, it says what counts as meeting it rather than describing the ideal, and it says when it does not apply. A fifth is the real proof: you can point at an awkward conversation and still read off an answer.
Turning vague criteria into checkable ones
The rewrite is less work than teams expect. You are not writing a policy document or padding the criterion for its own sake. You are replacing a word that names a quality with a description of the evidence that would demonstrate it. Here are five criteria that appear on almost every scorecard, why each falls apart under automated scoring, and a version that holds up.
| Vague criterion | Why it fails when automated | Rewritten version |
|---|---|---|
| Showed empathy | Empathy is a feeling, not a fact in the transcript. Two readers weigh warmth differently, so the same conversation scores both ways. | Acknowledged the customer’s stated problem or frustration in the agent’s own words before moving to a solution or a process step. |
| Was professional | A container word covering language, tone, patience and formality. Each reader unpacks it differently, and nobody can tell which part failed. | Used no profanity, sarcasm or dismissive language, and did not interrupt or talk over the customer while they were describing the issue. |
| Handled the issue well | Undefined, and it merges effort with outcome. An agent who did everything right on an unsolvable request scores like one who did nothing. | Either resolved the customer’s stated request within the conversation, or explained the specific next step, owner and timeframe before closing. |
| Used good judgment | Judgment is invisible. The reader has to reconstruct the decision faced and imagine the alternatives, which is guesswork. | Where policy allowed discretion, stated the option chosen and the reason for it, and did not commit to anything outside the documented policy limits. |
| Followed the correct process | A pointer to knowledge the criterion does not contain. Different readers hold different versions of the process in their heads. | Completed each required step for this issue type in order: verified identity, confirmed the account, recorded the reason code, and stated the resolution. |
Observable evidence beats inferred intent
The move behind every rewrite above is the same: replace what the agent meant or felt with what is present in the conversation. Intent has to be inferred, and inference is where two readers diverge. Evidence can be pointed at.
That is also what makes a score checkable afterwards. A score anchored to observable evidence traces back to the lines that produced it, so it can be verified, coached on, or overturned. One anchored to an inferred state can only be argued about, because neither side has anything to point at.
More rewrites, same move
- Before: Made the customer feel valued. After: Thanked the customer for their patience or for raising the issue at least once, and did not repeat a question the customer had already answered.
- Before: Took ownership. After: Used first person commitment language for the next action, such as stating what the agent will do, rather than describing what the customer must do or what another team may do.
- Before: Communicated clearly. After: Explained any technical term, policy name or internal reference the first time it was used, and confirmed the customer’s understanding before closing.
- Before: Set the right expectations. After: Stated a specific timeframe or next step for any action not completed during the conversation, rather than an open ended assurance such as soon or shortly.
- Before: Was efficient. After: Did not ask the customer for information already present in the ticket, the account record, or earlier in this conversation.
None of these rewrites run longer than a sentence or two, and none require knowledge the transcript does not contain. That is the bar.
Split compound criteria into single checks
Compound criteria are the most common structural fault in rubrics, and the easiest to fix. Greeted the customer and verified their identity looks like one line, but it holds two independent checks with one score attached. If the agent greeted warmly and skipped verification, what is the answer? Half? Pass, because most of it happened? Fail, because a compliance step was missed? There is no correct answer. Whatever the machine returns will be wrong for somebody, and the disagreement will look like a scoring error rather than a rubric error.
How to spot and split them
- Search for and, or, while, as well as, and mid-criterion commas. Almost every one hides a second check.
- Split into one criterion per check. Greeted the customer using the approved opening becomes one line, verified the customer’s identity using two approved identifiers becomes another.
- Weight them separately. This is the real prize. Verification is a compliance requirement and a greeting is a courtesy, so they should never have carried equal weight. Splitting lets you say so.
- Watch for hidden compounds. Resolved the issue and documented it correctly, answered accurately and quickly, and was polite and helpful are all two criteria wearing one coat.
Splitting also pays off in coaching: a compound criterion tells an agent they scored badly on a bundle, while two separate criteria tell them exactly which behavior to change.
Binary, scaled, and the not applicable case everyone forgets
Once a criterion checks one thing, decide what kind of answer it produces. Most rubrics reach for a five point scale by default, which is usually the wrong instinct.
Use binary when the behavior either happened or did not
Verification, required disclosures, reason codes and mandatory steps are yes or no questions. A scale invites the scorer, human or machine, to invent gradations that do not exist. Binary criteria produce the most stable agreement between a machine and a calibrated reviewer, and they are the backbone of a rubric that scores consistently at volume.
Use a scale only when the behavior genuinely has degrees
If you use a scale, define every point on it as an observable condition, so the difference between a two and a three is a fact rather than a feeling. A scale where only the top and bottom are described is a binary criterion with three undefined values in the middle, and those middles are where inconsistency collects. Three well defined points beat five vague ones.
Define not applicable, or it will quietly become a fail
This is the most commonly missed case in rubric design and the one that does the most damage to reported quality. Many criteria simply cannot apply to some conversations: no refund policy to explain when the customer asked about opening hours, no identity to verify on an anonymous pre sales question, no escalation path when nothing needed escalating.
With no explicit not applicable state, the scoring engine has two options and both are wrong. Marking it failed penalizes agents for the contact type they received. Marking it passed inflates scores and hides real misses. Either way your averages stop meaning anything, and the effect concentrates in whichever team handles the most unusual contacts.
The fix is to write the applicability condition into the criterion itself: where the customer requested a refund, the agent stated the refund policy and the expected timeframe. Not applicable if no refund was requested. Then confirm N/A criteria are excluded from the denominator in your reporting as well as your scoring. Check that directly: it is easy to configure correctly in the rubric and still get wrong in the average.
Writing the subjective criteria you cannot avoid
Some things you genuinely need to measure are subjective. Tone, empathy and rapport are often the difference between a technically correct conversation and a good one. The answer is not to drop them, nor to accept that they will score inconsistently forever. It is to anchor them to observable behavior rather than to the feeling they produce.
Ask what an agent actually does when being empathetic, and score that instead. In a transcript, empathy looks like acknowledging the problem before solving it, naming the impact on the customer, and not restating a policy at someone who is upset. Those are observable. The feeling is not.
Three subjective criteria, anchored
- Before: Maintained a positive tone. After: Did not use blaming or defensive phrasing, such as attributing the problem to the customer’s own actions, and did not respond to a complaint with policy language alone before acknowledging it.
- Before: Built rapport. After: Responded to at least one piece of personal context the customer volunteered, rather than moving straight to the next process step.
- Before: Showed confidence. After: Gave a direct answer or a clear next step without hedging language such as I think or it might be, and did not contradict an earlier statement in the same conversation.
Anchored subjective criteria will never reach the agreement rate of a binary compliance check, and neither do human reviewers, which is the real reason these criteria are hard. What anchoring buys is that disagreements become discussable. Two people arguing about whether an agent seemed warm get nowhere. Two arguing about whether the agent acknowledged the problem before quoting the policy can look at the transcript and settle it. For where automated scoring is strong and where it is not, see how accurate AI QA is.
Test a criterion before you deploy it
No criterion should go live on your whole conversation volume straight from a document. Testing costs an hour. Not testing costs a quarter of quality data nobody trusts.
A test that takes an hour
- Pick ten conversations on purpose, not at random: two or three where the criterion clearly applies, two or three where it clearly does not, and at least three awkward cases at the edges. The edges are where wording fails.
- Have two reviewers score it independently, from the wording alone, with no discussion first.
- Run the automated scoring over the same ten and compare all three sets of answers.
- Read the disagreements before you judge them. Where the two humans disagreed with each other, the wording is at fault. Where both humans agreed and the machine differed, look at the evidence the machine cited before assuming it was wrong: often it applied your words correctly and you did not mean them.
- Change the wording, not the verdicts, and run it again. Two or three passes is normal. A criterion needing a fourth is usually two criteria.
The step people skip is comparing against the machine’s cited evidence, and it gets skipped because most systems do not make it easy. If a score arrives as a number with no trace back to the lines that produced it, you cannot tell a rubric problem from a model problem, and every disagreement becomes opinion. Kaizo traces every score back to the specific evidence in the transcript, so a disputed score becomes a concrete question about wording rather than an argument about whether the AI can be trusted. There is a fuller walkthrough in how to validate AI QA scoring.
Maintaining a rubric without breaking your trend data
A rubric is not finished when it is deployed. Products, policies and customer language all change, so a criterion that was sharp in January can be quietly ambiguous by June. Rubric drift is normal. Unmanaged rubric drift is what makes a year of quality data unusable.
When to change a criterion
- Disputes cluster on it. One criterion generating a disproportionate share of challenged scores is a wording problem announcing itself.
- Reviewers disagree on it. Track which criteria your calibration sessions keep returning to.
- Its pass rate sits near 100 percent or near zero. A criterion almost nobody fails measures nothing, and one almost nobody passes is usually mis worded rather than describing a universal failure.
- The underlying policy changed and the criterion still describes the old one.
- Not applicable fires far more or far less than expected, which usually means the applicability condition is wrong.
Version, and note the date
Here is the part teams learn the hard way. If you edit a criterion in place, the scores before and after measure different things but sit in the same trend line looking comparable. A quality score that moves two points the week you tightened a definition reads as a performance change, and somebody will act on it.
The discipline is simple even if your tooling does not enforce it. Give each criterion a version and a change date. Never silently reword a live criterion: retire the old version and start a new one, so the break is visible in the chart. Record what changed in one line, and flag any trend that spans a rubric change rather than letting the reader assume the operation moved. Distinguish a clarification from a redefinition, too: fixing a typo does not change what is measured, but changing a threshold, an applicability condition or a weight does.
A checklist for every criterion you write
Run every criterion through this before it joins your live scorecard. A minute each removes most of the failure modes above.
- One check only. No and, no or, no commas hiding a second requirement.
- Named evidence. A reader knows exactly what in the transcript answers this.
- No inferred states. Nothing about how the agent felt, meant or seemed.
- A defined pass condition, not just a description of the ideal.
- An explicit not applicable rule, and confirmation that N/A is excluded from the average rather than counted as a fail.
- Every scale point defined, or the criterion converted to binary.
- No container words. Professional, appropriate, effective, proper, adequate and good all need unpacking.
- No external knowledge required. If it points at a process the reader must already know, bring the checkable part of that process into the criterion.
- Tested on ten conversations, including at least three edge cases.
- Versioned and dated, so a future change is visible in the trend.
One more check sits outside the rubric: ask who owns the system doing the grading. Most QA and CX platforms now sell their own AI agents, so their scoring engine is assessing conversations their own product handled. Kaizo does not sell AI agents, so it has nothing to defend in the conversations it grades, and at UiPath that neutral scoring automated 100% of QA with 200% ROI and an 8% lift in quality score. Scoring every conversation is the precondition rather than the achievement, and plenty of platforms now do it. What a carefully written criterion adds is the thing that makes coverage worth having: a score somebody can check.
If your automated scoring is misbehaving, this is the cheapest place to look first. See also the common auto QA mistakes that undo good rubrics.
Frequently asked questions
How do you write a QA criterion an AI can score consistently?
Write it so that two people, given only the transcript and your exact wording, would reach the same verdict. In practice that means naming the observable evidence in the conversation rather than a quality like empathy or professionalism, keeping one check per criterion, defining what counts as a pass rather than describing the ideal, and stating when the criterion does not apply. Then test the wording on ten real conversations, including edge cases, before you deploy it.
Why does AI QA score the same behavior differently on different conversations?
Usually because the criterion allows more than one reasonable reading, so different evidence in different conversations satisfies it. Ambiguous criteria do not fail loudly, they fail inconsistently, which looks like an unreliable model. Before assuming the scoring engine is wrong, check whether two of your own reviewers would agree on the same conversations using only the wording. If they would not, rewrite the criterion rather than changing the tool.
Should QA criteria be binary or scaled?
Use binary whenever the behavior either happened or did not, which covers most compliance and process criteria and produces the most stable agreement between a machine and a calibrated reviewer. Use a scale only when the behavior genuinely has degrees, and define every point on it as an observable condition. A scale where only the top and bottom are described is a binary criterion with undefined middle values, and those middles are where inconsistency collects.
How should not applicable be handled in an automated scorecard?
It needs to be an explicit state written into the criterion, and it must be excluded from the average rather than counted. Without it, a criterion that cannot apply either counts as a fail, penalizing agents for the contact type they received, or counts as a pass, inflating scores and hiding real misses. Write the applicability condition into the criterion itself, then verify that N/A is removed from the denominator in your reporting as well as your scoring.
Can subjective criteria like empathy be automated at all?
Yes, if you anchor them to observable behaviors instead of feelings. Score acknowledging the customer’s problem in the agent’s own words before offering a solution rather than showed empathy. Anchored subjective criteria will never match the agreement rate of a binary compliance check, and human reviewers do not match it either, but anchoring makes disagreements discussable against the transcript rather than a clash of impressions.
What happens to your quality trend when you change a criterion?
It breaks, silently, unless you version the change. Scores from before and after an edit measure different things but sit in the same trend line looking comparable, so a two point move caused by a tightened definition gets read as a performance change. Retire the old version and start a new one with a change date, note what changed in one line, and flag any trend that spans a rubric change.
Related terms
Test your rubric against real conversations
Bring your current scorecard and a set of conversations your reviewers have already scored. We will show you, criterion by criterion, where the automated score agrees, where it does not, and exactly which wording is causing the gap, with every score traced back to the evidence in the transcript. Kaizo does not sell AI agents, so it has nothing to defend in the conversations it grades.
