How to Build a Code Review Agent (Playbook)

Developers have a fast, decisive response to bad automated review: they mute it. A code review agent earns attention by finding real defects, citing the code, explaining how it fails, and staying quiet otherwise. This playbook covers building one that way, following FISTA's AI engineering practice. Context on AI in review is in ai code review and ai for engineering teams.

What does the agent do?

Review class	Method	Examples
Correctness	Model reasoning with context	Off-by-one, null handling, wrong branch, race conditions
Security	Static analysis plus model	Injection, auth bypass, secrets, unsafe deserialization
Standards	Linters and model	Error handling conventions, logging, API usage
Tests	Model with coverage data	Missing tests for changed behavior, weak assertions
Design	Model, advisory only	Duplicated logic, unclear boundaries

Step 1: Scope defect classes and standards

Decide what the agent reviews and what it leaves to humans and linters. Prioritize classes with high defect cost and detectability: correctness, security, and test gaps. Encode coding standards explicitly, referencing existing style and architecture documentation. Define prohibited behavior: no stylistic nitpicks that linters cover, no comments without a concrete failure scenario. See how to write an ai spec.

Step 2: Provide repository context

Diffs alone produce guesses. Give the agent:

• The full files touched, not just hunks.
• Definitions and call sites of changed functions, via a code index or language server.
• Repository conventions from documentation and configuration.
• Test files related to changed code and coverage data.
• Recent history of the touched files when relevant.

Context retrieval is bounded by token budgets and prioritized by relevance. Indexing patterns follow how to build a semantic search engine applied to code.

Step 3: Integrate static analysis

Run linters, type checkers, security scanners, and secrets detection first. Their findings are deterministic and precise; feed them to the agent as context so it does not duplicate them and can reason about their implications. The agent focuses on what static tools cannot: intent, behavior across calls, and missing cases.

Step 4: Generate findings with evidence

For each candidate finding, require:

1. Location: file and line range.
2. Claim: the specific defect.
3. Failure scenario: concrete input or state that produces the wrong outcome.
4. Confidence: a calibrated estimate.
5. Suggested fix where clear, marked as a suggestion.

Findings that cannot state a failure scenario are dropped. Structured output enforces the schema; see what is structured output.

Step 5: Control noise

• Confidence thresholds per defect class, set from evaluation.
• Deduplication of findings across similar locations.
• Suppression of classes already covered by linters.
• A cap on findings per review, ranked by severity and confidence.
• Developer feedback buttons (useful, not useful, false positive) that feed evaluation.
• Per-repository tuning of standards and thresholds.

Step 6: Evaluate precision and recall

Build a labeled set of real pull requests from your repositories: changes with known defects (from post-merge fixes and incident retrospectives) and clean changes. Measure:

Metric	Method
Precision by class	Share of findings that are real
Recall by class	Share of known defects found
Noise rate	Findings dismissed by developers
Severity accuracy	Ranking quality
Latency and cost per review	Traces

Precision is the adoption metric. Wire the suite into CI so prompt, model, or context changes are gated. Method is in the AI evaluation and testing whitepaper.

Step 7: Roll out advisory, then gate selectively

Start with advisory comments on pull requests, visible but non-blocking. Track acceptance, dismissal, and post-merge defects. Once a defect class shows sustained high precision, consider gating merges on that class alone. Never gate on classes with unproven precision; developers will route around it. See ai quality assurance.

Step 8: Security and data handling

Run the agent inside the CI environment under the same access controls as other tooling. Use approved provider terms or private deployment for code submission; see private llm vs public api. Scan for secrets before any submission. Log reviews for audit. Protect the agent against injection through code comments and commit messages; see prompt injection defense checklist.

What does it cost to run?

Cost scales with pull request volume, diff size, and context retrieved per review, and is controlled by context budgets, model routing, and caching of repository context. Measure cost per review against reviewer time and post-merge defect cost. Drivers are in llm api cost optimization.

What are the common mistakes?

• Reviewing diffs without surrounding context.
• Posting style comments that linters already make.
• Findings without failure scenarios.
• Gating merges before precision is proven.
• Evaluating on synthetic bugs instead of real pull requests.
• Ignoring developer feedback as evaluation data.

What does a good finding look like?

A useful finding reads like a colleague's comment, not a scanner's alert. It names the file and lines, states the defect in one sentence, gives the concrete input or state that triggers it, rates its confidence, and, where the fix is obvious, suggests it as a suggestion rather than a demand. For example: in the order-total function, when a discount code is applied to an empty cart the subtotal is zero and the percentage calculation divides by it, producing a not-a-number total that the checkout accepts; high confidence; consider guarding for an empty cart before applying percentage discounts. A finding that cannot be written in this form is not ready to post.

How FISTA Solutions builds code review agents

FISTA Solutions builds code review agents to this playbook: scoped defect classes and explicit standards, repository context via code indexing, static analysis integration, evidence-bearing findings with noise control, precision and recall evaluation on real pull requests, and advisory-first rollout. The AI enablement practice delivers the platform and CI integration, AI agents the reviewer itself, and forward deployed engineers embed with your engineering team to tune standards and thresholds. The approach reflects FISTA's own verification-led engineering across 150+ projects.

To scope a code review agent for your repositories, message FISTA on WhatsApp, or read ai test generation for the companion capability.