AI Penetration Testing: Scoping and Running Tests on LLM Systems

An LLM application passes a conventional penetration test with no critical findings and is then talked into revealing another customer's records by a sentence hidden in a support ticket. The conventional test never touched the model. AI penetration testing adds the model, its inputs, its tools, and its retrieval to the attack surface, and measures what an attacker can actually achieve through them. This guide covers scoping, attack classes, methodology, deliverables, and cadence, drawing on FISTA Solutions' AI agents practice. The adversarial companion is in the ai red teaming guide and the architecture being tested in the AI agent security architecture whitepaper. Testing must be authorized in writing and scoped to systems the organization owns or has permission to test.

What is in scope?

Surface	What is tested	Example
Inputs	Direct and indirect prompt injection through user input and content	Instructions hidden in a document the assistant summarizes
Retrieval	Permission bypass; extraction of restricted content	Querying for documents the user should not see
Tools and agents	Misuse, privilege escalation, gate bypass	Steering an agent to call a tool outside its authority
Outputs	Data exfiltration; harmful content; policy bypass	Coaxing memorized or retrieved secrets into responses
Memory and logs	Extraction from stores; unredacted sensitive data	Asking the agent what it remembers about other users
Gateway and APIs	Authentication, rate limits, key exposure, cost amplification	Bypassing the gateway to reach provider keys
Infrastructure	Conventional weaknesses in serving, vector stores, pipelines	Unauthenticated vector database endpoint
Supply chain	Model integrity, dependencies, plugins	Tampered model weights or malicious tool packages

Attack patterns in depth are in ai agent security risks and injection specifics in what is jailbreaking in ai.

How is the test scoped and authorized?

Define the systems, environments, and data in scope; obtain written authorization including for testing that reaches third-party model providers within their terms; use a test environment that mirrors production controls with synthetic or masked data where possible; agree rules of engagement for actions that could have side effects; and set the reporting and retest plan. Testing that skips authorization or hits production tools with live side effects is not testing.

What methodology works?

Start from the threat model and the system's controls; enumerate inputs the model reads, including indirect ones such as documents, tickets, emails, and web content; attempt injection through each to alter behavior, extract data, and drive tool calls; probe retrieval for permission bypass with identities of different scope; attempt tool misuse and gate bypass; test output channels and logs for leakage; assess the gateway, keys, rate limits, and cost amplification; and test infrastructure and supply chain conventionally. Record every attempt and outcome. Control design being tested is in ai and zero trust architecture.

How are findings rated?

By what an attacker could achieve: exposure of restricted data, unauthorized actions with financial or legal effect, bypass of safety controls reaching users, and cost amplification, weighted by ease and reach. A model that says something odd under contrived prompting is a low finding; an assistant that returns another tenant's records to a crafted query is critical. Ratings map to the risk register. Register practice is in ai risk register.

What should remediation look like?

Design changes, not prompt patches: structural separation of instructions from content, permission enforcement in retrieval and tools, validation of outputs and actions outside the model, least-privilege tool scopes, gates on consequential actions, redaction in logs, and gateway hardening. Each finding becomes an automated adversarial test case so it cannot regress. Defenses are in the prompt injection defense checklist and validation in llm output validation.

How does AI pentesting relate to red teaming and CI testing?

Penetration testing is scoped and methodical against defined systems; red teaming simulates realistic adversaries and campaigns more broadly; continuous adversarial suites in CI encode known attacks so every change is checked. Organizations run CI suites on every change, pentests before launch and after material changes, and red team exercises periodically by tier. The red team practice is in the ai red teaming guide and the concept in what is ai red teaming.

What cadence is appropriate?

Before launch for any system that touches sensitive data or takes actions; after material changes to models, prompts, tools, or retrieval; periodically by risk tier, often annually for high-tier systems; and continuously through automated suites. Provider model updates count as changes. Change control is in how to manage ai vendors.

Who should perform it?

Testers with both application security fundamentals and hands-on experience attacking LLM systems, internal or external, independent of the builders. The skill set is scarce, and organizations often combine an internal capability with periodic external assessments. Hiring guidance is in hire ai security engineers.

What mistakes weaken AI pentests?

Testing only direct user prompts and skipping indirect content; testing against a sandbox without production controls; rating findings by model misbehavior rather than attacker impact; remediating with prompt instructions; no retest; and no conversion of findings into CI tests. Each leaves the real risk in place.

What does a sound test look like in practice?

A support agent with retrieval over customer records and refund tools is tested before launch in a mirrored environment. Testers inject instructions through ticket content, attempt cross-customer retrieval with scoped identities, try to trigger refunds above threshold, and probe logs for unredacted data. Findings: one indirect injection path that alters tone, rated low; one retrieval filter gap for a document type, rated critical. Remediation fixes the permission sync, adds output validation, and encodes both attacks in CI. Retest confirms, and the system launches. The domain build is in how to build an ai customer service agent.

How FISTA Solutions approaches AI penetration testing

FISTA Solutions scopes and runs authorized AI penetration tests across inputs, retrieval, tools, outputs, memory, gateway, and infrastructure, rates findings by attacker impact, remediates through design changes, and converts findings into continuous adversarial suites. The AI agents practice delivers secured systems, AI enablement provides the gateway and testing platform, and forward deployed engineers embed with client security teams. The record behind the approach is 150+ projects with 99.9% uptime.

To find out what an attacker can do through your AI systems before one does, message FISTA on WhatsApp, or read the prompt injection defense checklist for the defenses a test will probe.