QA sampling means reviewing a subset of customer conversations, often two or three tickets per agent per week, instead of all of them. QA sampling exists because manual review is slow: it is a workaround for a capacity limit, and it is only as reliable as the sample is representative.
In short
- A sample gives a rough team read, too thin for individual agents.
- The hardest conversations are the least likely to be sampled.
- Automated scoring removes the capacity limit, so full coverage is replacing sampling for measuring agents.
- Sampling still works for checking that reviewers or graders score consistently.
What sampling means in support QA
Sampling in customer service QA is simply the decision to review some conversations and not others. Because a reviewer can only read so many tickets or listen to so many calls in a day, a QA team picks a subset, scores those against the scorecard, and uses the result to represent the whole. A common shape is a fixed number per agent per week, chosen to fit the reviewers’ available time.
The word to notice is represent. The sampled conversations stand in for all the ones that were not reviewed, and the entire value of the exercise depends on how well they do that. When the sample represents the whole well, sampling is a reasonable economy. When it does not, every conclusion drawn from it is built on conversations that were never actually looked at.
One clarification, because the term travels across industries: sampling here means selecting customer conversations to review. It is unrelated to statistical sampling in manufacturing quality control, which is a different practice with a different purpose.
Why teams sample, and what it costs
Sampling is a response to a hard constraint. If your team handles thousands of conversations a week and you have a couple of reviewers, reading everything is impossible, so you read a slice. There is nothing wrong with the logic. The problem is what the slice can and cannot support.
A small sample gives you a rough sense of how a team is doing overall, and for that it is adequate. It struggles the moment you use it to judge an individual, because a few conversations is a tiny and often unlucky window on someone’s month. Two agents of equal quality can look very different on a small sample purely because of which conversations happened to be pulled. There is also a selection problem: the conversations that most need review, the messy escalations and the ones that went quietly wrong, are often the least likely to be sampled, because reviewers gravitate to conversations that are quick to score. The full statistical treatment of this is in why a small QA sample cannot support agent-level decisions.
A worked example: what a weekly sample sees
Take a hypothetical agent who handles 100 email and chat conversations a week, with two of them sampled for review. The QA score for that week rests on 2% of the agent’s work. Over a month, eight conversations stand in for four hundred, and a single bad week can come from one unlucky pull.
Now look at what those eight conversations have to carry. If one is a quick password reset and another a long refund dispute, the score mostly reflects which tickets were drawn, not how the agent works. If the agent mishandled three escalations that month, there is a good chance none of them was in the sample, so the coaching conversation never happens. Raising the sample to five a week helps a little and costs a reviewer more hours, which is exactly the trade sampling was meant to avoid.
Sampling vs full coverage
Sampling reviews a subset sized to reviewer capacity, while full coverage scores every conversation. Sampling is adequate for a rough read on a team; full coverage is what agent-level decisions and trend detection need. The constraint that made sampling necessary is going away, because auto QA can now evaluate every conversation automatically, which reframes the choice.
Automated scoring, such as Kaizo AutoPilot, applies your own scorecard to all conversations rather than a chosen slice. If you are deciding how much to sample in the meantime, our guide to choosing a QA sampling plan compares the options side by side.
| Sampling | Full coverage | |
|---|---|---|
| What gets reviewed | A subset, sized to reviewer capacity | Every conversation |
| Good for | A rough read on a team | Agent-level decisions and trend detection |
| Blind spot | The conversations never pulled, including the worst | Requires trusting the automated scorer, so it must be verified |
| What the sample is for | Measuring quality | Checking the scorer, not measuring agents |
What teams see when they stop sampling
Teams that move from a manual sample to automated scoring report more coverage and less preparation work, not just bigger numbers.
At UiPath, Kaizo automated 100% of QA with 200% ROI and an 8% lift in quality score. The change was simple: the score stopped resting on a handful of conversations someone had time to read.
See it in a live demo. Kaizo scores 100% of your support conversations against your own scorecard and turns the findings into coaching. Let’s talk.
Where sampling still belongs
Full coverage does not make sampling useless; it moves sampling to a better job. When every conversation is scored automatically, you no longer sample to measure agents. You sample to check the measurement: to confirm that reviewers agree with each other and that an automated grader agrees with your reviewers, which matters more under automation, not less.
That is a legitimate, ongoing use of a sample, and it becomes more important under automation because a single grader now shapes every score. The related practices are QA calibration and validating AI QA scoring.
The move from sampling to coverage is also what lets QA stop being a source of argument on the floor. When an agent’s score rests on two conversations, it is easy to dismiss. When it rests on all of them, and every score traces back to the evidence, the conversation shifts from whether the sample was fair to what the conversations actually show. That is the case made in scoring 100% of conversations.
When QA sampling is enough
QA sampling is enough when the team is small, the volume is low, and the goal is a general sense of quality rather than decisions about individual agents. If a few agents handle a modest number of tickets a week, a lead can read a large share of them by hand, and a sample may already be close to full coverage.
Sampling is also a sensible first step for a team that does not have a scorecard yet. Automating the scoring of every conversation only helps once you know what you are scoring for, so building and calibrating the scorecard on a manual sample comes first. Once the team grows, or once scores start feeding coaching, performance reviews or bonuses, the thinness of the sample becomes the problem, and that is the point to look at auto QA.
Frequently asked questions
What is QA sampling in customer service?
It is the practice of reviewing a subset of customer conversations instead of all of them, typically a few per agent per period. It exists because manual review is slow and there are always more conversations than reviewers can read. The sampled conversations are treated as representative of the whole, so the method is only as good as how representative the sample actually is.
Why is QA sampling a problem for judging individual agents?
Because a few conversations is a tiny window on an agent’s month, and which conversations were pulled matters enormously. Two agents of equal quality can post very different sampled scores by chance, and the hardest conversations, which most need review, are often the least likely to be sampled. A sample sized for reviewer capacity is too thin to support a decision about one person.
How many conversations should a QA team sample?
There is no universal number. Most teams size the sample to reviewer time, often two or three conversations per agent per week, which is enough for a rough team read but not for decisions about one agent. If scores feed coaching or reviews, the sample has to be far larger, which is why teams move to scoring every conversation.
How should conversations be chosen for a QA sample?
Random selection keeps the sample closest to representative, because it stops reviewers from picking conversations that are quick to score. Some teams add targeted pulls, such as escalations or low customer ratings, to see the hardest cases. Keep the two apart when reporting, because a targeted pull is not representative of an agent’s overall work.
Is QA sampling the same as sampling in manufacturing?
No. In customer service, sampling means selecting conversations to review. It is unrelated to the statistical acceptance sampling used in manufacturing quality control, which inspects a subset of physical units against a defect tolerance. The two share a word but not a purpose.
Does full QA coverage replace sampling entirely?
It replaces sampling for measuring agents, because every conversation can be scored. It keeps sampling for a different job: checking the scorer, by confirming that reviewers agree with each other and that an automated grader agrees with your reviewers. That use of sampling becomes more important under automation, not less.







