How to Run Shadow-Mode Deployments for AI Agents

Specifications are written from documents, and documents describe the process as it was meant to run. Shadow mode is where the agent meets the process as it actually runs: the tolerances people quietly apply, the exception nobody wrote down, the data field that means something different in one region. This playbook runs shadow mode so that those discoveries happen before launch, at no risk, and become specification decisions with evidence. It is step six of how to onboard a Digital FTE and applies the evaluation-driven development whitepaper to live traffic.

Step 1: What are the prerequisites?

A specification and a golden dataset that the agent already passes; integration through the tool layer with read-only permissions for shadow mode; the baseline instrumented; a comparison store separate from any system of record; and the process owner's commitment to triage disagreements on a cadence. Permissions design for the read-only phase is in how to design tool permissions for AI agents.

Step 2: How is the shadow path built?

Live inputs are duplicated to the agent as they arrive; the agent runs its full logic including tool reads but every write is redirected to the comparison store as a proposal; the existing process produces the real outcome; a matcher pairs proposals with outcomes by case identifier. Nothing the agent does reaches a customer, a record, or a queue. Traces are captured for every case.

Component	Purpose
Input tap	Delivers live cases to the agent without affecting routing
Proposal store	Records what the agent would have done, with the trace
Outcome capture	Records what the existing process did
Matcher and scorer	Pairs and scores agreement by category
Triage queue	Presents disagreements to the process owner with both sides and the trace
Dashboard	Agreement by category over time; disagreement distribution

Step 3: What is compared?

For decisions: did the agent reach the same classification, routing, or resolution as the person. For actions: would the proposed end state match the actual end state. For escalations: did the agent escalate the cases people treated as exceptions, and only those. Score by category, because aggregate agreement hides the category that fails.

Step 4: How are disagreements triaged?

Finding	Meaning	Action
Agent wrong	Model, prompt, or retrieval issue	Fix; add case to the golden dataset
Person wrong	Practice diverged from policy	Process finding for the owner; case may still join the dataset with the policy outcome
Specification silent	The case was never covered	Write the rule; add cases; re-run
Data problem	Input or reference data wrong	Upstream fix; note as a dependency
Human-only	Judgment the agent should not make	Add to non-scope; escalation rule

The "specification silent" row is usually the largest at the start and should shrink each cycle. The "person wrong" row is a gift: it reveals control gaps the business did not know it had.

Step 5: How do revision cycles work?

Weekly, or more often at high volume: review the disagreement distribution with the process owner, decide the actions, update the specification and dataset, run the regression gate, redeploy the agent to the shadow path, and continue. Track agreement by category across cycles; the trend is the evidence.

Step 6: When does shadow mode end?

When agreement per category is stable at a level the process owner accepts, remaining disagreements are understood and either fixed or assigned to non-scope, and the shadow report is complete. Not when a date arrives. If agreement plateaus below the acceptable level after revisions, that is a stop condition for the project, as described in the AI project charter template.

Step 7: What does the shadow report contain?

Volume by category; agreement by category over cycles; the final disagreement distribution; specification changes made; process findings; recommended autonomy level at launch with the evidence; and the oversight capacity the exception rate implies. It is the launch decision's evidence and the first performance review's baseline.

Step 8: How does launch follow?

Launch at the recommended level, usually suggest or act-with-approval, with the shadow path retained for a period so agreement can still be measured on cases people handle. The autonomy ladder is described in Digital FTE performance review.

What does shadow mode look like week to week?

Week one: agreement on the standard path is high, agreement on exceptions is poor, and half the disagreements are "specification silent"; the process owner writes four rules nobody had documented. Week two: exception agreement rises; a cluster of "person wrong" cases reveals that one region applies a tolerance the policy does not allow, and the owner escalates it as a control finding. Week three: agreement is stable across categories; the remaining disagreements are a judgment category the owner assigns to non-scope with an escalation rule. The report recommends launch at act-with-approval with a staffed exception queue sized from the observed exception rate.

What about workflows with low volume?

Low-volume workflows reach statistical confidence slowly, and the temptation is to launch on a handful of agreements. Two techniques help: replay historical cases through the shadow path to widen the sample, clearly labeled as replay rather than live, and extend the shadow period rather than lower the agreement bar. If the workflow is too rare to evaluate, it is usually too rare to justify an agent, and the shadow report should say so.

What are the common mistakes?

1. Write permissions in shadow mode.
2. Aggregate agreement hiding a failing category.
3. Disagreements untriaged, so nothing improves.
4. Ending on a date.
5. No process owner in the loop.
6. Skipping shadow mode because the dataset passed.

How does FISTA Solutions help?

FISTA Solutions runs shadow mode on every AI agent it deploys, with forward deployed engineers building the shadow path and triaging disagreements alongside the process owner, and the AI enablement practice making the shadow infrastructure reusable across agents. FISTA has delivered 150+ projects for 50+ companies across 12+ countries with 99.9% uptime.

To run shadow mode for an agent you are about to launch, message FISTA on WhatsApp, or read how to build a golden dataset for the artifact shadow mode feeds.