Enterprise RAG Reference Architecture: A Whitepaper

Retrieval-augmented generation is the most common architecture for enterprise LLM applications, and the most commonly under-engineered. The typical first build is a vector index over a folder of PDFs and a prompt that says "answer using the context." It demos well and fails in production: wrong documents retrieved, permissions ignored, stale content served, confident answers with no basis. This whitepaper describes an enterprise RAG reference architecture that treats retrieval as the system it is.

What is retrieval-augmented generation, and why does the architecture matter?

RAG grounds a language model's answers in retrieved enterprise content instead of relying on the model's training data. The model is the last step in a pipeline; everything before it determines whether the answer is correct, authorized, and current. The basics are in what is RAG, and the failure modes in why RAG systems hallucinate. This whitepaper assumes that foundation and focuses on the production design.

What are the layers of the reference architecture?

Layer	Responsibility	Key decisions
Sources and connectors	Pull content from systems of record	Which sources, change detection, auth
Ingestion and normalization	Parse, clean, structure, enrich	Parsers per format, metadata extraction, PII handling
Chunking	Split into retrievable units	Strategy, size, overlap, structure awareness
Indexing	Store chunks for lexical and vector retrieval	Embedding model, vector store, lexical index, metadata schema
Permissions	Attach and enforce access control	Entitlement model, query-time filtering
Query processing	Understand and transform the question	Rewriting, decomposition, routing
Retrieval	Find candidate chunks	Hybrid search, filters, top-k
Reranking	Order candidates by relevance	Cross-encoder or LLM reranker, cutoff
Generation	Produce a grounded answer	Prompting, citations, refusal, output validation
Evaluation	Measure retrieval and answer quality	Labeled sets, metrics, regression gates
Operations	Keep it fresh, fast, affordable, observable	Refresh schedules, caching, tracing, cost controls

How should ingestion and normalization be designed?

Ingestion is where most quality is won or lost. Enterprise content is heterogeneous: PDFs with tables, slide decks, wiki pages, tickets, emails, spreadsheets, and database rows. Each needs a parser that preserves structure (headings, tables, lists) and extracts metadata (source, author, date, document type, business unit, access control lists, version). Three rules:

1. Preserve structure. Tables flattened to text lose meaning; headings dropped lose the context that makes chunks answerable.
2. Extract metadata aggressively. Metadata powers filtering, permissions, freshness, and citations. A chunk without provenance is unciteable.
3. Handle sensitive data at ingestion. Redact or tag PII and secrets before they reach the index, according to policy. See AI data leakage prevention.

Change detection matters as much as initial load. Connectors should support incremental updates and deletions, because a deleted document that remains in the index is both a quality and a compliance problem.

What chunking strategy should enterprises use?

Chunking determines what a retriever can find. Fixed-size chunking is easy and usually wrong for enterprise documents. Better strategies:

• Structure-aware chunking that respects headings, paragraphs, and table boundaries.
• Hierarchical chunking that indexes small chunks for precision but returns their parent section for context.
• Overlap to avoid splitting a fact across boundaries, tuned per document type.
• Chunk enrichment, prepending document title, section path, and key metadata to each chunk so it is self-describing when retrieved.

There is no universal chunk size; the right one is found by evaluation against your own question set. Concepts are covered in what is chunking in RAG.

How should indexing be designed?

Enterprise RAG indexes twice: a vector index for semantic similarity and a lexical index for exact terms, identifiers, product codes, and names that embeddings handle poorly. The embedding model is chosen by evaluation on your content and pinned; changing it requires reindexing and a regression run. The vector store choice, whether a dedicated database or an extension to an existing one, is an operational decision driven by scale, filter complexity, and team familiarity. Guidance is in how to choose a vector database and pgvector vs dedicated vector database. The metadata schema is designed up front, because it drives filtering and permissions.

How are permissions enforced?

Permission enforcement is the difference between an enterprise system and a liability. The rule is simple: the model must never see content the user is not authorized to see. Implementation:

1. At ingestion, attach the source system's access-control information to every chunk (groups, roles, sensitivity labels).
2. At query time, resolve the requesting user's entitlements from the identity provider.
3. Apply entitlements as a hard filter in retrieval, before ranking, so unauthorized chunks are never candidates.
4. Log which chunks were retrieved for which user for audit purposes.
5. Re-sync entitlements on a schedule and on change events, because access changes.

Relying on the prompt to tell the model not to reveal certain content is not a control. See AI access control.

How should queries be processed before retrieval?

User questions are rarely good search queries. A query-processing stage improves retrieval substantially:

• Query rewriting to expand abbreviations, add context from conversation history, and normalize phrasing; see what is query rewriting.
• Decomposition of multi-part questions into sub-queries retrieved separately.
• Routing to the right index or source when the enterprise has several, or to a structured query path when the answer lives in a database rather than documents.
• Intent classification to detect questions the system should refuse or escalate.

Why hybrid retrieval and reranking?

Hybrid retrieval combines vector and lexical results, typically with reciprocal rank fusion, so semantic matches and exact-term matches both surface. On enterprise content, with its part numbers, policy names, and acronyms, hybrid consistently beats pure vector search in evaluation. Reranking then applies a more expensive model, a cross-encoder or an LLM, to the top candidates to order them by true relevance to the question, and cuts the list to what fits the context budget. The combination is the single highest-leverage improvement most first-generation RAG systems can make. Background is in what is hybrid search, what is BM25, and what is a reranker.

How is grounded generation enforced?

Generation is constrained so that the answer is traceable to evidence:

Control	Purpose
Citation requirement	Every claim references a retrieved chunk; uncited claims are flagged or removed
Refusal on insufficient evidence	If retrieval returns nothing relevant above a threshold, the system says so rather than guessing
Context discipline	Only reranked, authorized chunks enter the prompt, within a token budget
Output validation	Structure, citation validity, and policy checks run before the answer is shown
Groundedness scoring	An automated check that the answer is supported by the cited chunks, sampled in production

These controls are what make a RAG answer defensible in a regulated setting. See how to ground an LLM and what is groundedness in AI.

How is RAG evaluated?

Evaluate retrieval and generation separately, because they fail differently and are fixed differently.

• Retrieval metrics: recall at k and precision at k against a labeled set of questions mapped to the chunks or documents that answer them. Low recall means chunking, indexing, or query processing problems.
• Generation metrics: answer correctness against reference answers, groundedness against retrieved context, citation accuracy, and refusal correctness on unanswerable questions.
• End-to-end: human-rated quality on a stratified sample, and user feedback in production.

The labeled set is built with domain experts from real questions, versioned, and extended with production failures. Every change to chunking, embeddings, retrieval, reranking, prompts, or model reruns the suite. Method detail is in the AI evaluation and testing whitepaper and how to improve RAG accuracy.

What operational concerns does the architecture address?

• Freshness: incremental ingestion with change detection, deletion propagation, and a visible index-age indicator per source.
• Latency: caching of embeddings and frequent queries, parallel retrieval across indexes, streaming generation, and a latency budget per stage.
• Cost: reranker cutoffs, context budgets, model routing, and caching, tracked per query.
• Observability: traces covering query, rewritten query, retrieved and reranked chunks, prompt, answer, citations, latency, and cost, with sampled quality scoring.
• Multi-tenancy: index or namespace isolation and tenant-aware filtering where the system serves multiple business units or customers.

When should the architecture extend to Graph RAG or agentic RAG?

Two extensions address specific limits. Graph RAG adds a knowledge graph of entities and relationships so questions that span many documents or require multi-hop reasoning can be answered; it suits domains with rich structured relationships. Agentic RAG lets an agent plan multiple retrieval steps, choose sources, and verify intermediate results; it suits complex research tasks at the cost of latency and complexity. Neither replaces the base architecture; both sit on top of it. See what is graph RAG, how to build a graph RAG system, and how to build an agentic RAG system.

What does a phased build look like?

1. Foundation: connectors for two or three high-value sources, structure-aware ingestion, metadata schema, permissions model, hybrid index.
2. Quality: labeled evaluation set, reranking, query rewriting, grounded generation with citations and refusal.
3. Operations: incremental refresh, observability, cost controls, dashboards, runbooks, owner.
4. Expansion: more sources, routing, structured-data paths, and where justified, graph or agentic extensions.

Each phase ends with an evaluation report, not a demo.

How FISTA Solutions builds enterprise RAG

FISTA Solutions builds RAG systems to this reference architecture as part of its AI enablement practice: permission-aware ingestion, hybrid retrieval with reranking, grounded generation, and a retrieval-and-generation evaluation harness that runs on every change. Forward deployed engineers work with your content owners and security team to get the metadata, entitlements, and labeled question set right, which is where enterprise RAG succeeds or fails. The same foundation powers the AI agents we deploy on top of it. The record behind the method is 150+ projects delivered with 99.9% uptime.

To assess an existing RAG system against this architecture or to scope a new one, message FISTA on WhatsApp, or continue with the step-by-step guide how to build a RAG system.