Evaluation-Driven Development for AI Systems: A Whitepaper

Most AI projects that stall share a symptom: nobody can say, with evidence, whether the system is good. Demos look convincing, stakeholders disagree about individual outputs, and every change is a gamble because there is no way to know whether it made things better or worse. Evaluation-driven development is FISTA's answer to that problem. It treats the definition of correct as the first deliverable, the evaluation harness as the release gate, and production quality as a measured quantity.

This whitepaper is written for engineering leaders, product owners, and the risk functions that need evidence before approving automation. It covers the principle, the artifacts, the scoring methods, the release gates, the production loop, and the organizational arrangement that keeps it working. It expands on LLM evaluation explained and the AI evaluation and testing whitepaper with the end-to-end method.

Why must evaluation come first?

Probabilistic systems do not fail the way deterministic software fails. A prompt change can improve one category of case and quietly degrade another; a model upgrade can raise average quality and introduce a new failure mode; a retrieval tweak can fix hallucinations on one document set and cause them on another. Without a fixed reference and an automated way to score against it, none of those effects is visible until users find them.

Evaluation first inverts the usual sequence:

Sequence	Build first	Evaluate first
Definition of correct	Emerges from demos and disagreements	Written down and agreed before building
Quality signal	Anecdotes and stakeholder opinion	Scores against a golden dataset
Change safety	Every change is a gamble	Every change is gated on regression
Launch decision	Confidence	Evidence against thresholds
Production quality	Unknown until complaints	Sampled and scored continuously

The discipline is the software-engineering expression of the principle behind spec-driven development: if a task cannot be specified well enough to score, it is not ready to be delegated.

What are the artifacts?

Evaluation-driven development produces four artifacts, each with an owner.

1. The correctness specification: the criteria an output must satisfy, written in language a domain expert and an engineer can both verify. It includes hard constraints (never do X), required content, format, tone where it matters, and the conditions under which the system must abstain or escalate.
2. The golden dataset: representative inputs paired with verified expected outcomes, organized by category, difficulty, and failure mode. Building it is described in how to build a golden dataset.
3. The scoring harness: code that runs the system against the dataset and scores each output against the criteria, producing per-category results and a comparison with the previous run.
4. The production sampling pipeline: a process that draws real traffic, scores it with the same criteria, and routes misses back to the dataset and to the process owner.

These artifacts are versioned together. A change to the correctness specification changes the dataset and the harness, and the change is reviewed like code.

How should outputs be scored?

Use the cheapest method that is reliable for each criterion, and combine methods per output.

Criterion type	Method	Notes
Exact or structural	Deterministic checks: schema validation, field equality, regex, numeric tolerance	Fast, cheap, unambiguous; use wherever possible
Retrieval quality	Precision and recall against labeled relevant documents	Isolates retrieval from generation
Groundedness	Claim-by-claim check against provided sources, mechanical or model-based	Central for RAG systems; see why RAG systems hallucinate
Subjective quality	Rubric-based model judge calibrated against human ratings	Publish agreement rate; recalibrate when models or rubrics change
Safety and policy	Classifiers plus rule checks for prohibited content and actions	Zero-tolerance criteria are hard gates
Task success (agents)	End-state verification: did the ticket get created with the right fields	Score outcomes, not transcripts

Model-based judges are useful and dangerous. They scale human judgment, but they inherit model biases and drift with model versions. Calibrate them on a human-rated sample, report agreement, and re-check the calibration whenever the judge model or the rubric changes. The pattern is described in LLM evaluation explained.

How do release gates work?

Every change that can affect output quality goes through the harness before release: prompt edits, model upgrades, retrieval changes, tool changes, and configuration changes. The gate compares the new run with the baseline and blocks release when:

• Any zero-tolerance criterion fails on any example.
• Aggregate quality falls below the threshold the process owner set.
• Any category regresses beyond its tolerance, even if the aggregate improves.
• Cost or latency per case exceeds budget.

Category-level gates are the important detail. Aggregate scores hide the case where a change helps common cases and hurts rare, expensive ones. The gate design is expanded in AI regression testing, and its place in the delivery pipeline is described in the agentic SDLC whitepaper.

How does the production loop work?

Launch is not the end of evaluation; it is the point where the dataset starts learning from reality.

1. Sample production traffic continuously, stratified by category and by confidence, with heavier sampling where the system is least certain.
2. Score the sample with the same harness and criteria; where a criterion needs human judgment, route to reviewers with a queue and a service level.
3. Route misses to the process owner for triage: is this a spec gap, a dataset gap, a model weakness, or a data-quality problem upstream?
4. Add verified misses to the golden dataset so the failure cannot recur silently.
5. Track drift: category mix, input distribution, and score trends over time. Rising misses in a stable spec usually mean the world changed. See what is model drift.
6. Feed the numbers into autonomy decisions: the evidence that justifies moving an agent from suggest to act is exactly this data.

The human review queue that powers steps two and three is described in how to build a human review queue.

How do you evaluate agents that take actions?

Evaluating a classifier or a summarizer means comparing an output with a reference. Evaluating an agent means checking what it did, which requires a different setup.

• End-state verification. Run the agent in a sandboxed copy of the systems it touches, then inspect the resulting state: was the ticket created with the right fields, was the record updated correctly, was nothing else changed. Transcript quality is secondary to outcome correctness.
• Tool-call traces. Score the sequence of tool calls against the spec: required calls made, prohibited calls absent, parameters valid, approvals requested where the rules demand them. A correct outcome reached by a prohibited path is still a failure.
• Multi-step task suites. Golden examples for agents are scenarios, not single prompts: a starting state, a request, and the expected end state, including cases where the correct behavior is to stop and escalate.
• Adversarial cases. Include scenarios where retrieved content contains injected instructions, where data is missing, and where the request is out of scope, and score whether the agent declines correctly.
• Cost and step budgets. Record steps, tokens, and latency per scenario; a change that keeps outcomes but doubles steps is a regression.

The harness for this is described in how to build an agent evaluation harness. The organizational lesson is the same as for simpler systems: define the expected end state with the process owner before building, because that definition is the specification.

What does the organization have to change?

Evaluation-driven development fails when it is treated as an engineering task. It works when three roles are explicit.

Role	Responsibility
Process owner	Owns the correctness specification and the thresholds; adjudicates misses; decides autonomy levels
Engineering	Owns the harness, the gates, the sampling pipeline, and the evidence reports
Reviewers	Score the human-judgment sample; their agreement rate calibrates the judges

Two rules protect the integrity of the loop. Engineering cannot lower a threshold to ship; only the process owner can, and the change is recorded. And the golden dataset is a shared asset with a change process, not a file on an engineer's laptop.

What are the common failure modes?

1. Evaluating after building. The dataset is assembled from whatever the system happens to do well.
2. Aggregate-only gates. Rare categories regress unnoticed.
3. Uncalibrated judges. A model grades another model and nobody checks it against humans.
4. Static datasets. Production misses never make it back; the dataset drifts away from reality.
5. Scoring transcripts instead of outcomes. An agent that sounds right but creates the wrong record passes.
6. No owner for the definition of correct. Disputes about quality are settled by whoever argues loudest.

How does FISTA Solutions apply evaluation-driven development?

Evaluation-driven development is how FISTA Solutions builds every AI agent and AI system it delivers. Our forward deployed engineers start engagements by writing the correctness specification with your process owners, building the golden dataset and harness before the first prompt is tuned, gating every change on regression, and standing up the production sampling loop before handover. The AI enablement practice installs the method across teams so it survives the first project. It is the discipline behind FISTA's record of 150+ projects for 50+ companies with 47% average efficiency gains: outcomes that were measured, not asserted.

If your AI systems are being judged by demos rather than data, talk to FISTA on WhatsApp about an evaluation assessment, or start with how to build a golden dataset.