AI Regression Testing: Gating Every Change on Evidence

Conventional regression testing asks whether a change broke something that used to work. AI regression testing asks the same question about a system whose behavior is probabilistic, sensitive to prompts and models, and dependent on retrieval and tools that change on their own schedules. The answer is the golden dataset run on every change with category-level gates. This guide describes the practice as FISTA applies it, implementing the evaluation-driven development whitepaper and supporting the release discipline in the agentic SDLC whitepaper.

Why is regression different for AI systems?

Property	Deterministic software	AI system
Change surface	Code	Code, prompts, models, retrieval, tools, configuration, dependencies
Failure shape	Errors and wrong outputs, reproducible	Quality shifts by category, often without errors
Detection	Unit and integration tests	Golden dataset scored by criteria, by category
Upstream change	Rare	Model providers update, indexes drift, tools change
Variance	Low	Real; must be measured, not ignored

A prompt tweak that improves the common case can degrade the rare one; a model version bump can change tool-call formatting; a re-embedding can shift retrieval. None throws an error. Only the suite sees it.

What triggers a run?

Every change in the classes above, in CI, before merge or deployment. Scheduled runs also catch upstream drift: nightly on the production configuration, and on notice of any provider model update. Runs are cheap relative to incidents and should never be skipped for schedule.

How are gates defined?

Gate	Rule
Zero-tolerance	Any failure blocks: prohibited actions, safety criteria, data-handling rules
Category threshold	Each category must meet its absolute threshold
Category regression	No category may drop more than its tolerance versus the baseline, even if aggregate improves
Agent end state	Every scenario's end state must verify; tool-call sequences must respect the specification
Budgets	Cost per task and latency percentiles within limits

Thresholds and tolerances are set by the process owner in the AI evaluation plan template.

How is variance handled?

Score against criteria, not exact strings. Run cases multiple times where the criterion is sensitive to variance and report pass rates and distributions. Prefer deterministic scoring; where a judge is needed, fix the judge model version, calibrate against human ratings, and re-calibrate when either changes. Treat a regression as a meaningful drop, defined per category, not a single differing output. Judge design is discussed in LLM evaluation explained.

What does the regression report contain?

Baseline and candidate versions; results by category with thresholds and deltas; zero-tolerance results; end-state results for scenarios; cost and latency against budgets; cases that changed outcome, with traces; and the gate decision. The report is attached to the release and, for routing changes, to the routing rule.

What is the release policy?

Failing gates block. Waivers are granted only by the process owner, in writing, with reason, expiry, and a compensating control (reduced autonomy, increased sampling). Engineering cannot waive its own gates. Waivers are a tracked metric and are reviewed at the next performance review. Agents cannot modify the suite or the gates for the change they implement, per the AI-generated code security checklist.

How does regression testing connect to migrations and routing?

The same suite qualifies replacement models, gates routing changes, and canaries migrations, as described in how to migrate between LLM providers and how to design a model routing strategy. A team with a regression suite treats a deprecation as scheduled work; a team without one treats it as an emergency.

How does the suite stay relevant?

Production misses join the golden dataset; retired categories are removed; coverage is reviewed quarterly. A suite that has not changed in six months is testing last year's work. The maintenance loop is in how to build a golden dataset.

What does a regression cycle look like?

An engineer changes the retrieval chunk size to improve answers on long policies. The suite runs in CI: the long-policy category improves, the short-FAQ category drops by more than its tolerance because chunks now carry irrelevant neighbors, and cost per answer rises with the larger context. The change is held; the engineer adds a length-aware chunking rule; the suite passes with both categories above threshold and cost within budget; the report is attached to the release. Two weeks later a scheduled nightly run flags a drop in groundedness after a provider model update, and the routing rule is switched to the tested fallback until the new version is re-qualified.

What are the common mistakes?

1. Aggregate-only gates.
2. Skipping runs for "small" prompt edits.
3. Judges unpinned and uncalibrated.
4. Engineering waiving its own gates.
5. No scheduled runs, so upstream drift is found by users.
6. A stale suite.

How does FISTA Solutions help?

FISTA Solutions installs regression gates as part of the evaluation platform in its AI enablement practice and ships every AI agent with its suite wired into CI, with forward deployed engineers running the first cycles alongside your team. FISTA has delivered 150+ projects for 50+ companies across 12+ countries.

To put regression gates on systems that ship without them, message FISTA on WhatsApp, or read the AI evaluation and testing whitepaper for the full testing model.