This AI evaluation sampling guide shows how to pick the samples that actually tell you something: how many you need, how to cover the hard cases, and why a clever small batch beats a lazy big one.
We grade outputs. We do not train models, run fine-tuning, or sell experts we cannot verify.
AI evaluation sampling is the method for choosing which model outputs to grade. You cannot grade everything, so you pick a batch that represents real usage: the common cases, the edge cases, and the cases you are most worried about. Good sampling means your grades generalize to the outputs you did not grade. Bad sampling means you measured the easy cases and learned nothing.
Most teams do not have a sampling method. They have a folder of whatever was handy.
Samples cherry-picked from demos and happy paths. They grade beautifully and tell you nothing, because the model already handled these. A batch of easy wins is a press release, not an evaluation.
The last fifty outputs, grabbed from yesterday's logs. Recent is convenient, but last week might have been unusually quiet, unusually weird, or dominated by one power user. Convenience is not representativeness.
All samples from English, or from short queries, or from the simplest task type. The model looks great on the slice you measured and falls apart on the slices you skipped. Coverage across your real input mix is the whole job of sampling.
Eight samples. Even if they are well chosen, eight samples cannot show you a pattern; they show you anecdotes. You need enough samples for the repeating failures to repeat.
There are two honest ways to pick a batch, and most good evaluations use both. Here is the difference in plain terms.
Pull outputs at random from your real logs. Every output has an equal chance of being picked. The upside is honesty: no human bias in the selection, and the results reflect your actual input mix. The downside is that rare but important cases might not show up. If 2 percent of your traffic is medical questions and those are the ones that can hurt people, a random draw of 50 might include one, or none.
Split your outputs into groups first, then sample from each group on purpose. The groups, called strata, are the dimensions that matter: task type, input length, language, user segment, topic area. You decide how many samples each group gets. This guarantees coverage of the slices you care about, including the rare dangerous ones. The cost is that you have to think: choosing the strata is itself a judgment call about what matters.
Start stratified. List the dimensions where your product could fail differently: the task types, the languages, the tricky topics, the edge-case inputs you already worry about. Allocate samples to each, weighting toward the dimensions with the most risk, not the most traffic. Then add a small random draw on top to catch whatever your strata missed. A 50-sample batch built this way tells you more than 500 samples grabbed from yesterday's logs. For the full evaluation this sampling feeds into, see our LLM evaluation guide.
The honest answer: enough for patterns to repeat, few enough that grading stays careful. Here is how to think about it.
For a first evaluation, 20 to 50 samples is the sweet spot. Below 20, you get anecdotes. Above 50, grading cost climbs faster than insight for most products. In our pilots, the repeating failure patterns almost always show up by sample 30. If your product has many distinct task types, go toward 50 and spread them across the strata.
High-stakes outputs, many languages, or a dozen task types can push a thorough evaluation past 50. The rule stays the same: cover every stratum with enough samples to see a pattern inside it. Five samples per stratum is a reasonable floor. If the math says 80, grade 80, but grade them in batches so reviewers stay sharp.
A narrow product doing one thing, like summarizing support tickets, can learn plenty from 20 well-chosen samples. Narrow scope means fewer strata, which means fewer samples to cover them. Do not pad the batch to hit a number; pad the coverage to hit the risk.
One batch is a snapshot. Models drift, prompts change, users find new ways to break things. Re-run your sampling on a schedule, monthly is a common cadence, and compare severity distributions over time. That is the difference between an evaluation and continuous monitoring.
You send 20 to 50 model outputs plus your rubric, or we help you write one. We can help you pick the batch too.
Trained reviewers grade every sample against the rubric, backed by automated checks.
Sample-level grades, issue patterns, recommended fixes, and a walkthrough call.
Yes, when you have them. Real user inputs beat anything you invent in a conference room. Strip anything sensitive first, then stratify by the dimensions that matter. If you are pre-launch, build the batch from realistic synthetic inputs that mirror what users will actually ask.
Ask where the model could fail differently. Task type almost always matters. Then consider input length, language, topic sensitivity, and any segment your business cares about most. Three to six strata is typical. If you list twelve, merge the ones where failures would look the same.
Yes, deliberately. If you already know a class of bad outputs, include examples so the grading confirms how bad they are and whether your fixes worked. A batch with zero known failures is either a great product or a batch that avoided the hard questions.
Include a small set if safety matters for your product: jailbreak attempts, prompt injections, requests for disallowed content. These belong in their own stratum so they do not distort the quality grades. Our red teaming service covers this in depth.
Yes. In a pilot, we work with you on the batch before grading starts: reviewing your strata, checking coverage, and flagging gaps. Good sampling is half the evaluation, so we treat it as real work, not an afterthought.
Keep the strata and the rubric stable, then compare severity distributions batch to batch: the share of P0s and P1s should fall as you fix things. Change the rubric mid-stream and you are measuring two different things. Stability in method is what makes trends trustworthy.
Send 20 to 50 samples. We will help you pick them, then grade every one.
Not sure where to start? Talk to us and we will point you at the right evaluation.