RAG in Production 2026: The Enterprise Team Checklist

"The demo was great. In production the AI answers wrong, slowly, or not at all." I hear that sentence regularly since 2024 — most recently on a CompanyGPT project where half the test questions still failed after three weeks of prompt tuning. Cause: not the model, but retrieval, chunking, and missing access control.

I come from enterprise search — ten years of Sinequa, indexes, ACLs, relevance tuning. For me, RAG isn't a new invention; it's search plus LLM. Teams that ignore search discipline and only tweak prompts don't build production systems. This article is my checklist for teams getting serious after the proof of concept.

At Join GmbH in Eisenach I work exactly at this intersection: CompanyGPT for internal knowledge work, with the same demands for access, relevance, and traceability I had in search projects before. The difference: the LLM formulates the answer — but only as well as retrieval delivered.

Context Without Hype

2024 was the year of RAG demos: upload PDF, ask question, applause. In 2026, operational quality decides. Hallucinations, slow answers, and data leaks aren't "AI quirks" — they're architecture bugs that can be fixed.

Production RAG means documented chunking strategies, hybrid search, index-level ACLs, evaluation before every change, cost control, and observability. That sounds like work because it is work. But it's predictable work — not months of guessing at the system prompt.

1. Chunking with Intent, Not Defaults

A default of 512 tokens for everything rarely suffices. I always test chunking with 20–30 real questions from the business unit — not "What's in chapter 3?"

Embedding model: Changing the embedding model means a full re-index. I document model choice and dimensions before the first index run. Mixing old and new embeddings in the same index isn't an option.

2b. Parsing Before Chunking

Before chunks exist, the text must be right. PDFs with scanned pages need OCR. Tables in Word or PDF need layout-aware extraction — otherwise cell values land in the wrong order in the chunk. That's the most common reason "the AI explains the table wrong": it never saw the correct table.

4b. Regression on Every Change

Chunking, embedding model, top-k, reranker — every change can improve faithfulness and hurt relevance. So I measure at least three metrics at once. A pipeline with 95% faithfulness but 40% relevance answers honestly — but often "no information," because the wrong chunks arrive.

2. Hybrid Search — Not Optional

Pure vector search fails on exact terms: product codes, legal paragraphs, ticket IDs, serial numbers. I combine:

• Dense (embeddings) for semantic similarity
• Sparse (BM25/keyword) for exact matches
• Reciprocal rank fusion or weighted scores

In Sinequa and modern stacks, hybrid is often standard. For greenfield projects on Qdrant or Elasticsearch, I plan hybrid from day one — not as a later upgrade.

3. ACL at Index Level

RAG without ACL is a data leak. Every query must:

1. Know user identity (SSO, token)
2. Apply filters at index level — not prompt-only post-filter
3. Show only sources the user may read in the answer

Post-filter alone isn't enough. LLM context must never contain chunks the user can't see — even if the answer doesn't cite them. That's not a new rule from my search years, but RAG projects overlook it surprisingly often.

4. Evaluate Before Scaling

I build a golden set with 30–50 real questions — from the business unit, not invented by IT:

• Faithfulness: answer only from context?
• Relevance: correct chunks retrieved?
• Latency: p95 under your SLA?

Tools like RAGAS or DeepEval help; manual labels in a spreadsheet suffice early on. Critical: measure before and after every pipeline change. Otherwise you don't know whether new chunking helped or hurt.

I store golden-set questions in a sheet: question, expected source, actual source, faithfulness (yes/no), comment. After four weeks the team has a shared picture — not just "feels better."

5. Cost Control

RAG can get expensive — not from the chat model, but embeddings and volume:

• Embedding cache for unchanged documents
• Smaller models for routing and classification
• LiteLLM budgets per team
• Limit top-k (5–8 chunks often enough)
• Re-index only on real document changes, not weekly "just in case"

6. Observability

No logs, no debugging when "the AI lies." Per request I log:

• Query (anonymized if needed), retrieved chunk IDs, model, tokens, latency
• User feedback (thumbs up/down)
• Errors, timeouts, empty retrieval results

For pilots in Open WebUI, a simple request log often suffices. In production, this belongs in central monitoring.

6b. Re-Index and Document Lifecycle

Deleted documents must leave the index — not only at the next big-bang re-index. Changed versions need new chunks and invalidation of old ones. I plan from day one: event from CMS/DMS → queue → embedding → upsert/delete. Without this process the AI answers with outdated prices, old contacts, or policies that no longer exist. That's not a hallucination problem — that's a stale index.

6c. Model Change Without Surprises

A new chat model can often be switched via LiteLLM alias. A new embedding model means re-index. I keep both in separate runbooks — otherwise the team thinks an alias change is enough and wonders why relevance dropped.

7. Answers with Sources

Users trust answers with citations — and lose trust without them. My UI pattern:

• Inline references [1], [2]
• Click opens source document at the right location
• Allow and encourage honest "I don't have information on that"

An AI that always answers is more suspicious than one that shows limits.

Agents and MCP — Where RAG Ends

For tool access and IDE integration I use MCP — but MCP doesn't replace a RAG pipeline. Knowledge questions over documents go through retrieval with ACL. Actions in systems (create ticket, commit code) go through tools with their own permissions. Architecture must separate the two.

What I Do Differently in Projects

Retrieval before prompt. The first two weeks go to chunking, hybrid search, and golden set — not system prompt wording.

One document type at a time. Manuals first, tickets later, email maybe never. Each type needs its own chunking.

Pilot with real users early. Ten people from the business unit beat 100 internal IT tests.

No big-bang re-index. Incremental updates with a clear strategy for deleted and changed documents.

Automation for ops, not knowledge questions. Recurring processes (summary, classification) go to n8n. Open knowledge questions go to RAG with sources.

One line I often say in reviews: if your golden set isn't larger after four weeks than on day one, you're not testing — you're hoping. Every real user feedback item belongs in the set, anonymized if needed. That's how a demo becomes a learning system.

Production Checklist

1. Chunking strategy per document type documented and tested
2. Hybrid search enabled
3. ACL at index level implemented and verified with test accounts
4. Golden set with 30–50 questions and metrics (faithfulness, relevance, latency)
5. LiteLLM or comparable cost governance
6. Request logging and feedback loop
7. Sources in the UI, including "no information"
8. Runbook for re-index, model change, and provider outage

Conclusion

Production RAG is enterprise search plus LLM — not "ChatGPT with PDF upload." Teams that combine search discipline with LLM governance build CompanyGPT that business units trust. The rest is tuning.

Support with RAG architecture and rollout? LinkedIn