How to Build a Golden Dataset for AI Evaluation

Every evaluation, gate, and performance review in FISTA's method rests on one artifact: the golden dataset, the verified definition of correct that a system is measured against. Built carelessly, it tests what the system already does well and misses the cases that cause incidents. Built deliberately, it makes quality a measurement. This playbook builds it deliberately. It implements the evaluation-driven development whitepaper and feeds the AI evaluation plan template.

Step 1: What does correct mean?

Write the correctness criteria with the process owner before collecting a single case: per output, what must be present, what must never appear, the format, and the conditions under which the system must abstain or escalate. Each criterion names a scoring method. The criteria decide what the dataset must contain; the dataset cannot be designed without them.

Step 2: How should coverage be designed?

Dimension	Design
Categories	Every case type the system handles, with target counts per category
Difficulty	Easy, typical, hard, and edge cases within each category
Failure modes	Known weaknesses represented explicitly, so regressions are caught
Out of scope	Cases where the correct behavior is to decline or escalate
Adversarial	Injected instructions, misleading content, missing data
Ambiguity	Cases where the correct behavior is to clarify
For agents	Scenarios: starting state, request, expected end state, expected tool calls and approvals

Write the coverage plan as a table with target counts and sourcing per cell before collecting.

Step 3: Where do cases come from?

Real historical cases, redacted, with consent and privacy review, are primary because they carry the actual distribution and quirks. Sample stratified by category rather than by recency. Synthetic cases fill documented gaps (rare failure modes, adversarial inputs) and are labeled as synthetic. Production misses join continuously after launch. Privacy handling follows the AI privacy impact assessment checklist.

Step 4: How are outcomes verified?

Domain experts verify each expected outcome against the criteria; consequential categories get two-person verification; non-obvious cases get a written rationale. Disagreements between verifiers are findings about the specification and are resolved by the process owner. Unverified cases are excluded, not included with a caveat. The verification load is real work and belongs in the project plan.

Step 5: What does a case record contain?

Field	Content
Identifier and version	Stable identifier; dataset version added
Category, difficulty, source	For stratified scoring and provenance
Input	The redacted input or, for agents, the starting state and request
Expected outcome	The verified correct output or end state
Scoring method	Deterministic check, rubric, end-state test
Rationale	Why this is correct, where not obvious
Verifier and date	Provenance
Tags	Failure mode, adversarial, synthetic, production miss

Step 6: How is the dataset versioned and governed?

Store it in version control with the system; every change is a reviewed pull request with a rationale; dataset versions are tied to release evaluations; the process owner approves additions and removals; and nobody edits expected outcomes to make a run pass. The specification and the dataset change together.

Step 7: How does it grow from production?

Sample production outputs stratified by category and confidence; route misses to the process owner for triage; add verified misses and their variants; retire cases that no longer reflect the work. Track the share of production categories represented and the age of the dataset's newest cases. The loop is described in how to build a human review queue.

Step 8: How is it used?

By the evaluation harness on every change, producing results by category against thresholds; by production sampling with the same criteria; by performance reviews; and by model selection and migration as the vendor-neutral benchmark, per how to choose an LLM for enterprise agents.

How are scoring methods attached to cases?

Each case carries the scoring method its criterion needs, and the harness applies it without human judgment where possible. Structural and exact-value criteria use deterministic checks: schema validation, field equality, numeric tolerance. Retrieval criteria use labeled relevant sources and compute precision and recall. Groundedness uses claim-by-claim comparison against the sources the case names. Subjective criteria use a rubric judge whose agreement with human raters has been measured on a calibration slice of the dataset and is re-measured whenever the judge model or the rubric changes. Agent scenarios use end-state verification against a sandboxed copy of the systems involved, plus checks on the tool-call sequence. Cases whose criterion cannot be scored mechanically are flagged for the human review sample, and their count is kept small on purpose, because a dataset that needs people to score it will not be run on every change. The scoring choices are documented in the AI evaluation plan template.

What does the dataset look like in daily use?

An engineer changes a prompt to handle a new document layout; the harness runs the full dataset and reports that the new layout category passes but a rare exception category regressed by two cases; the change is held until those are fixed. A reviewer in the weekly sample finds a case the agent resolved incorrectly; the process owner confirms the correct outcome, and the case and two variants are added with a rationale. At the quarterly review, the dataset's coverage table shows one production category under-represented, and collection for it is scheduled. At migration time, the same dataset scores the replacement model, and its category-level results decide which workloads move.

What are the common mistakes?

1. Collected by convenience, so it tests the easy cases.
2. Unverified outcomes.
3. No adversarial or out-of-scope cases.
4. Static after launch.
5. Expected outcomes edited to pass a run.
6. Owned by engineering, so correct means whatever ships.

How does FISTA Solutions help?

FISTA Solutions builds the golden dataset with process owners as the first deliverable of every AI agent engagement, through forward deployed engineers, and delivers it as a buyer-owned artifact alongside the harness, with the practice installed across teams through AI enablement. FISTA has delivered 150+ projects for 50+ companies across 12+ countries.

To build the dataset for your next system, message FISTA on WhatsApp, or read how to build an agent evaluation harness for the tooling that runs it.