A short trial can show whether an AI tool feels promising. It cannot show whether the tool will remain useful across repeated work, changing inputs, known failure modes, and real review conditions.
A small evaluation set closes that gap. It is a compact collection of representative tasks with fixed inputs, expected evidence, acceptance criteria, and review notes. The goal is not to create a universal benchmark. The goal is to make one team's decision more repeatable and less dependent on memory, demos, or a single impressive answer.
NIST's testing and evaluation work emphasizes meaningful tasks, context, measurement limits, and continued evaluation. The AI RMF also separates mapping the intended use from measuring performance and deciding whether deployment should proceed. NIST AI 800-3 adds an important warning: results on a fixed benchmark are not automatically estimates of performance on every similar future task. A small evaluation set should therefore support a bounded decision, not a claim that a tool is generally best.
1. Define the decision the evaluation must support
Start with the decision, not the test prompts.
Write one sentence such as:
Decide whether this tool should enter a four-week pilot for drafting support tickets that a human reviewer approves before sending.
A useful decision statement names:
- the repeated job;
- the intended users;
- the decision or output being produced;
- the reviewer;
- the allowed data;
- the next stage that a pass would permit.
Do not use a vague goal such as "evaluate writing quality." It leaves the evaluator free to change the standard after seeing the output.
A pass from a small evaluation set should normally allow only a longer pilot. It should not approve production, remove human review, or authorize broader data access.
2. Sample the real work distribution, not the easiest examples
Collect examples from work that actually occurs. Avoid selecting only clean, short, familiar tasks.
A small set of 12 to 30 items can be useful for initial screening when it covers distinct classes of work. The number is not a statistical guarantee. Coverage matters more than a decorative sample size.
Include at least four groups:
| Group | Purpose | Example |
|---|---|---|
| Common case | Measures routine usefulness | A typical customer question with complete context |
| Difficult case | Tests ambiguity or long context | A request containing conflicting requirements |
| Failure case | Exposes a known risk | A document with an unsupported factual claim |
| Refusal or escalation case | Tests boundaries | A request requiring legal, security, or manager review |
If the workflow serves several languages, departments, document types, or user roles, include them explicitly. Do not assume one English example represents the entire workload.
Record why each item belongs in the set. This prevents later replacement of difficult items with easier ones.
3. Freeze inputs, context, configuration, and test date
The same prompt can produce different results when the model, plan, system instructions, retrieval source, tools, temperature, or conversation history changes.
For every run, record:
- provider and product;
- model or model family shown in the interface;
- plan and account type;
- enabled tools and connectors;
- system or workspace instructions;
- input files and source versions;
- test date and region;
- whether the run was first attempt or retry.
Store the input separately from the output. A screenshot alone is weak evidence because it may hide the exact prompt, omitted context, or configuration.
When provider settings cannot be fixed, mark the run as partially controlled. Do not compare it as if it were identical to a deterministic software test.
4. Write acceptance criteria before running the tool
Each item needs criteria that can be checked without inventing a standard after the answer appears.
Use a mix of deterministic and human-reviewed checks.
Deterministic checks may include:
- required fields are present;
- prohibited fields are absent;
- dates and totals match a source;
- links resolve to approved domains;
- the output follows a schema;
- no confidential identifier appears.
Human-reviewed checks may include:
- the answer addresses the actual user request;
- the tone is appropriate;
- uncertainty is visible;
- the recommendation follows the supplied evidence;
- the reviewer can approve the result without reconstructing missing context.
Keep the criteria short enough that two reviewers can apply them consistently.
5. Separate critical failures from quality preferences
A minor style problem should not receive the same treatment as a privacy breach or fabricated citation.
Classify outcomes before testing:
| Severity | Meaning | Example | Default action |
|---|---|---|---|
| Critical | Unsafe or unauthorized result | Sensitive data exposed or action taken without approval | Block |
| Major | Result cannot be approved without substantial rework | Unsupported factual claim in a final recommendation | Fail item |
| Minor | Usable after limited correction | Formatting or tone issue | Record correction time |
| Preference | Subjective improvement | Alternative phrasing | Do not count as failure unless required |
This separation prevents a high average score from hiding one unacceptable failure.
A tool that passes 19 easy items and leaks confidential data on one item should not be described as having a 95 percent success rate without highlighting the critical failure.
6. Run repeats and keep the complete output record
Generative outputs can vary. Run each high-risk or unstable item more than once when practical.
Preserve:
- every raw output;
- tool calls or citations shown;
- latency and visible errors;
- retry count;
- reviewer edits;
- final decision for the item.
Do not keep only the best result. Selective retention converts evaluation into marketing.
For initial screening, two or three repeats on the most important items can reveal instability. A larger decision may require more runs and a formal analysis. The evaluation set should state its limits instead of pretending a few repeats measure all future variation.
7. Compare against a baseline and measure accepted outcomes
The relevant question is not whether the AI output looks sophisticated. It is whether the workflow improves on the current alternative.
Use a baseline such as:
- the existing manual process;
- a simpler model;
- a template without AI;
- the current approved tool;
- no automation.
Measure at least:
- acceptance rate;
- critical and major failure count;
- reviewer correction time;
- total completion time;
- cost per accepted outcome;
- unresolved uncertainty;
- user or reviewer preference where relevant.
A faster first draft may still lose if reviewers spend more time verifying it. A higher-quality model may still be unsuitable if it cannot meet data or permission requirements.
8. Record uncertainty and avoid universal conclusions
A small evaluation set supports a narrow statement such as:
On 20 selected support-drafting cases from July 2026, configuration A produced 15 outputs accepted with minor or no edits, two major factual failures, and no observed critical data-handling failures.
It does not support:
Configuration A is 75 percent accurate for customer support.
The second statement generalizes beyond the sampled items and may hide differences in task difficulty. NIST AI 800-3 distinguishes performance on a fixed benchmark from performance across a broader population of similar items. Small internal sets should make this boundary explicit.
Record:
- which work types were not sampled;
- which risks were not measured;
- which settings were uncontrolled;
- how many repeats were run;
- what evidence would change the decision.
Small evaluation-set template
| Field | What to record |
|---|---|
| Item ID | Stable identifier |
| Work class | Common, difficult, failure, escalation |
| Input version | Exact prompt and file hashes or versions |
| Expected evidence | Source, calculation, schema, or reviewer requirement |
| Critical rule | Condition that blocks adoption |
| Major criteria | Conditions required for approval |
| Run configuration | Model, plan, tools, date, settings |
| Raw result | Complete unedited output |
| Reviewer result | Pass, minor, major, critical, unresolved |
| Edit time | Minutes required for approval |
| Notes | Failure pattern and follow-up action |
When to refresh the set
Refresh the set when:
- the model or product changes materially;
- prompts, retrieval, tools, or system instructions change;
- a new failure appears in production or pilot use;
- the work distribution changes;
- policy, privacy, or approval requirements change;
- reviewers begin memorizing the answers instead of evaluating behavior.
Keep retired items when they represent important regressions. Add new items without silently deleting difficult historical cases.
Short checklist
Before trusting a small evaluation set, confirm that:
- the decision and next stage are defined;
- real common and difficult work is represented;
- at least one failure and escalation case is included;
- criteria were written before outputs were reviewed;
- critical failures are separated from preferences;
- raw outputs and retries are preserved;
- a current baseline is measured;
- the conclusion states its sample and uncertainty limits;
- the set has an owner and refresh trigger.
A useful evaluation set is not large because it contains many prompts. It is useful because each item represents a real decision boundary and produces evidence that can be reviewed again.