QA score traceability means every deduction in a scored conversation can be traced back to four things: the criterion that was applied, the exact moment in the conversation that triggered it, the version of the rule it was judged against, and the named person who can overturn it. It is not an explanation of how the model works. It is the record a score leaves behind, and it is what lets someone who was not there check whether the score was right.
In short
- A total with no breakdown is not a score, it is a verdict. It cannot be checked, coached against, or disputed.
- Four fields make a deduction traceable: the criterion, the evidence, the rule version in force at the time, and the decision of record.
- Traceability is not explainability. Explainability describes the model. Traceability describes the decision, against a rule a human published.
- A fluent model rationale is not evidence. Evidence is a pointer into the transcript, not a paraphrase of it.
- Criteria that cannot be evidenced cannot be traced, so traceability is a rubric design constraint before it is a software feature.
- Coverage is now table stakes across the category. What you can show behind a score is the part that is still hard.
Why a total score with no breakdown is unusable
Here is the moment every automated QA programme eventually reaches. An agent opens their scorecard, sees 70 out of 100, and asks one question. Which thirty points did I lose, and where?
If the only thing your system can produce is the total, no amount of grading accuracy saves you. Three things happen, in this order, and they happen every time.
The conversation stops being about the behaviour. Instead of discussing the moment the agent gave the customer the wrong refund window, the one to one becomes a discussion about whether the tool is any good. That is a worse use of a manager’s hour and it teaches the agent nothing.
The score becomes uncoachable. You cannot practise a total. You can practise acknowledging a stated problem before offering a solution. A number with no attached moment gives a person nothing to change on Monday.
The programme loses the benefit of the doubt. Once an agent has been told a number they cannot interrogate, every future number arrives pre-doubted. Trust is not lost in a dramatic incident. It leaks, one unexplained deduction at a time, and it is the fastest way to lose the ability to run a QA programme agents trust.
Manual review had traceability almost by accident. A reviewer wrote a comment, and if the comment was thin you could walk over and ask what they meant. Software does not have a desk. Unless the record is designed deliberately, automation deletes the explanation that used to come free, and the route for challenging a score has nothing to operate on.
Traceability, accuracy and disputes are three different problems
These get collapsed into one conversation about trusting the AI, and they should not be, because they are solved by different work and they fail independently.
- Accuracy asks how often the grader is right. A statistical property, measured in aggregate against reviewers who agreed with each other first.
- Validation asks whether the grader is right on average, and proves it. A test you run on a frozen reference set, criterion by criterion, on a schedule. The method is set out in how to validate AI QA scoring.
- A dispute is the process for challenging one specific score that one specific person believes is wrong. A workflow with owners, timeframes and outcomes.
- Traceability is the evidence layer underneath all three. Not a measurement, not a process. It is what the system stored at the moment it made the decision.
The dependency runs one way. You cannot validate what you cannot inspect, and you cannot dispute what you cannot see. A system can be highly accurate and still be untrustworthy in practice, because accuracy nobody can check on their own case is indistinguishable from a confident guess. Traceability is what turns a claim about accuracy into something an agent can verify for themselves, on their own conversation. It is also the cheapest piece to specify, and everything else in the verification cluster stands on it. This is not a support-specific idea. NIST’s AI Risk Management Framework exists to help organisations build trustworthiness into how AI systems are designed, used and evaluated, and none of that is checkable without a record of what the system decided.
The four things every deduction has to carry
A traceable deduction is not a longer comment. It is a record with four fields, each of which passes a specific test. Miss one and the record degrades in a predictable way.
| Field | What it holds | The test it has to pass | What breaks without it |
|---|---|---|---|
| Criterion | The exact rubric line that was applied, worded the way the agent has already seen it published | The agent can match the deduction to a line in the scorecard they were given on day one | The dispute widens from one criterion to the whole score, because there is nothing narrower to argue about |
| Evidence | A pointer into the conversation: the message, the turn, the timestamp, the words themselves | A reviewer who was not involved can find the moment in under a minute, without reading the whole thread | Every check becomes a full re-read, and the review queue you automated away quietly reforms |
| Rule version | Which version of the scorecard, criterion wording and threshold was in force when the score was produced | Rescoring the same conversation under the same version returns the same answer today | Scores change meaning retroactively, and no month is comparable with any other month |
| Decision of record | Who confirmed or overturned the deduction, when, and the reason they gave in their own words | You can list every overturned deduction from last quarter and read why each one was overturned | Overrides happen informally, nothing feeds back into the rubric, and the same error repeats next month |
Traceability is not explainability, and the difference matters
When people ask for explainable AI scoring they think they are asking for one thing. They are asking for two, and only one of them is yours to give.
Explainability, in the machine learning sense, is about the model. Which inputs moved the output, how much weight sat where, what the decision boundary looks like. A legitimate field, and largely irrelevant to the agent standing in front of you. Telling someone how a language model weighted their tokens is not more respectful than telling them nothing, it is just less honest about the fact that you are not answering the question.
Traceability is about the decision. Which published rule was applied, to which moment, producing which outcome, changeable by whom. It is a record rather than an interpretation, and it is within your control regardless of what model sits underneath.
The practical consequence is worth writing down: an agent is owed the rule and the evidence, not the model.
Why a generated rationale is not evidence
This trap is new, so it is worth naming precisely. When a language model grades a conversation it can also produce a paragraph explaining itself. That paragraph is fluent, confident, and reads exactly like reasoning. It is not evidence. It is generated text produced alongside the score, and it can be perfectly plausible while describing something the agent never said.
Evidence has to be a pointer, not a paraphrase. “The agent did not verify identity” is a restatement of the deduction. “At 14:32 the agent shared the last four digits of the card, and the identity check at 14:29 was not completed” is evidence, because it can be wrong in a way you can catch. If a rationale cannot be resolved to a specific turn in one click, treat it as commentary and keep it out of an agent’s scorecard. This is one of the sharper limits of using a model as a judge, and it is a design problem rather than a model problem.
How to test your own traceability in an afternoon
You do not need a vendor to tell you where you stand. You need ten conversations and about two hours, and this runs against a manual programme as easily as an automated one.
Pull ten recently scored conversations that lost points. For every deduction, answer these five questions using only what the system already stored. No asking the reviewer, no reconstructing from memory.
- Which criterion was applied? Named, and matching the published wording rather than an internal shorthand.
- Where in the conversation did it happen? A specific message or timestamp you can click to, not a summary of the whole interaction.
- What was the rule at the time? The threshold and wording that were live on the date of the score, not today’s version.
- Who could overturn this, and how would that be recorded? A named role, and a place the reason gets written down.
- Could you reconstruct all four answers in six months? After the rubric has been revised twice and the reviewer has left.
Score each deduction out of five. The result is usually uncomfortable in the same place: most programmes do well on question one, badly on question two, and have never once thought about question three.
The two failures this drill reliably exposes
Silent versioning. Somebody edited a criterion in March, and every score before and after it now sits in the same table looking comparable. It is not. Version the scorecard, date the version, and never recalculate a historical score under new wording. An agent’s March score was produced under March’s rules and should stay that way permanently.
The undocumented override. A team lead disagrees with a deduction, quietly adjusts the number, and tells the agent verbally. Everyone behaves well and the programme still loses, because the most valuable signal you have about your rubric, a case where a human thought the rule was wrong, evaporates. Overrides are not admissions of failure. They are the input to your next calibration session, and only if the reason is captured at the moment it is felt.
What an audit trail actually has to contain
Teams tend to assume they already have an audit trail because the system writes logs. A log records what the software did. An audit trail records what was decided, by whom, against which rule. They are not the same artefact, and the second one is the one anybody reviewing your programme will ask for. Regulators have already written down what they expect a record to do: Article 12 of the EU AI Act requires high-risk AI systems to log events automatically across their lifetime specifically so the system’s behaviour can be traced afterwards. Support QA is not in scope of that article, but the shape of the obligation is the right shape to copy.
A decision record that holds up needs six properties.
- Append-only. Corrections are added as new entries, never written over the old one. If the record can be edited in place, it is a document, not a trail.
- Time-stamped at decision time. Not at export time, and not at the time somebody noticed.
- Attributed to an actor. A named person or a named system component. “Updated by admin” is the answer that ends the review badly.
- Bound to a rule version. The specific criterion wording and threshold in force, carried with the decision rather than looked up later.
- Complete on overrides. Original value, new value, who, when, and the reason in free text. The reason is the part people skip and the part that carries all the learning.
- Reconstructable and exportable. Someone with no context can rebuild how a specific score came about, and get it out of the system in a readable form.
Two habits matter more than the storage design. Decide your retention period deliberately with whoever owns customer data protection at your company, and make it match what you tell agents, because a trail that has been silently pruned is worse than no trail. And treat a request to inspect the record as routine rather than adversarial. Programmes where asking to see the evidence is a normal Tuesday do not develop the resentment that programmes where it is an escalation always do.
Design the rubric so it can be evidenced
Here is the part most teams discover too late. Traceability is not something you add to a scoring system. It is a constraint on what you are allowed to put in the scorecard in the first place, because a criterion that cannot point at anything can never be traced, no matter how good the tooling is.
Use one test on every criterion. Try to write this sentence out loud:
“You lost X points because at [moment] you [did or did not do specific thing], and the criterion says [published wording].”
If you cannot complete that sentence with a real conversation in front of you, the criterion is not scoreable, it is an impression. Rewrite it or remove it.
Two versions of the same intention
- Not evidenceable: “Showed empathy.” There is no moment to point at. Two reviewers will disagree and neither can be shown to be wrong.
- Evidenceable: “Acknowledged the problem the customer stated before offering a solution.” There is a specific turn where this either happened or did not.
- Not evidenceable: “Handled the account securely.”
- Evidenceable: “Completed identity verification before discussing account details.” Two turns, in a checkable order.
You lose nothing real in this translation. The second wording in each pair is what the first was always trying to mean, and it is now coachable, disputable and consistent across reviewers. Running the test across a whole scorecard usually removes two or three criteria and rewrites half the rest. Build the constraint in from the start while designing the rubric rather than retrofitting it after the first round of disputes, and if a model will be doing the grading, write it as a rubric built for AI scoring from the outset.
One caveat, so this does not go too far. Not every criterion needs a single quotable line. Some are legitimately about the shape of the whole conversation, such as whether the agent repeated a question the customer had already answered. Those can still be evidenced, they just point at two moments instead of one. The test is not brevity, it is whether a specific part of the transcript can be produced.
What changes when the AI scores every conversation
The principles above are the same whether a person or a system produces the score. Full coverage changes two things, and both of them raise the cost of a thin record.
Volume. With 100% coverage revealing trends that 3% QA sampling never could, the number of deductions your programme generates rises by more than an order of magnitude. An unevidenced deduction was an annoyance at forty a month. At thousands, it quickly becomes the defining experience agents have of QA.
Consistency cuts both ways. A machine applies the same interpretation every time, which is the entire point. It also means an ambiguously worded criterion produces the same false positive across every conversation it touches, in the same direction, silently. A human reviewer would have drifted and hedged. The system will not, so the only thing standing between a bad criterion and a systematic problem is being able to look at the evidence behind a pattern of deductions and notice they are all wrong in the same way.
That is the argument for treating the evidence layer as the priority rather than the coverage number. Scoring every conversation instead of a sample is now the ordinary expectation in this category, claimed by more or less every vendor in it. Being able to show why a specific deduction happened is not.
Kaizo’s Auto QA scores against the criteria in your own scorecard and leaves the reasoning attached to the conversation, so a disputed deduction can be opened, traced to the exchange that caused it, and overturned with the reason recorded, rather than argued from memory in a one to one.
The test for any setup, yours or a vendor’s, is the same and it takes one question. Pick a conversation that lost points, ask which points and why, and see whether the answer is a rule and a moment, or a number and a shrug.
Frequently asked questions
What is QA score traceability?
QA score traceability is the ability to trace every deduction in a scored conversation back to the criterion that was applied, the moment in the conversation that triggered it, the version of the rule in force at the time, and the named person who can overturn it. It is a property of the record a score leaves behind, not of the model that produced it. A system with high accuracy and no traceability still cannot answer the only question an agent asks, which is which points they lost and where.
What is the difference between traceability and explainability in AI QA?
Explainability describes the model: which inputs moved the output and how the system arrived at a prediction. Traceability describes the decision: which published rule was applied, to which moment in the transcript, producing which outcome, and who can change it. An agent is owed the rule and the evidence, not the model internals. Traceability is also within your control, because it is a record you design, whereas explainability depends on what the model exposes.
Why is a total QA score not enough?
Because a total cannot be checked, coached against or disputed. An agent told they scored 70 will ask which thirty points they lost, and if the system returns only the total, the discussion becomes about the tool rather than the behaviour. A total also hides whether one criterion is failing everywhere, which is the signal you need to catch a badly worded rule before it distorts a quarter of scores.
Can an AI explain why it gave a QA score?
It can produce a rationale, but a generated rationale is not evidence. Language models write fluent, confident explanations that can describe things the agent never said. Treat model-written reasoning as commentary until it resolves to a specific message or timestamp. The usable standard is that every deduction points at a moment a human can open and read, so the claim can be checked rather than believed.
What should an AI QA audit trail contain?
Six things: an append-only record, a timestamp taken at decision time, a named actor for every entry, the rule version in force when the score was produced, complete override records including the reason in free text, and the ability for someone with no context to reconstruct and export how a specific score came about. A log of software events is not an audit trail, because it records what the software did rather than what was decided and against which rule.
How do you make QA criteria traceable?
Test each criterion by completing this sentence with a real conversation open: you lost X points because at this moment you did or did not do this specific thing, and the criterion says this. If you cannot finish it, the criterion is an impression rather than a scoreable behaviour. Replace it with an observable one. Showed empathy becomes acknowledged the problem the customer stated before offering a solution, which points at a specific turn and can be shown right or wrong.
Related terms
See the evidence behind a score, not just the number
Bring ten conversations your team has already scored. We will walk through what sits behind each deduction, and what your current setup can and cannot show an agent who asks which points they lost.
