What a Great RAG Pipeline Actually Looks Like

If you've spent time with RAG systems, you know the gap between a RAG pipeline that looks good in a demo and one that actually works in production is enormous.

The demo version: user asks a question, the system retrieves some chunks, the LLM produces a fluent, confident-sounding answer. Impressive in a slide deck. Quietly catastrophic when a user asks something where the relevant information isn't in the top-k results — or worse, when two contradictory documents are both retrieved and the model splits the difference.

Here's what separates a production-grade RAG pipeline from one that only works when you're watching.

Retrieval is where most pipelines fail

The most common mistake is treating retrieval as a solved problem once you've set up a vector store. It isn't.

Dense retrieval (pure embedding similarity) works well when the question is semantically similar to the relevant passage. It fails badly when the question is specific and literal — exact product codes, names, numbers — because those don't embed in a way that captures their specificity.

Hybrid retrieval (BM25 + dense, re-ranked) handles a much wider range of queries. It's not glamorous, but it's the difference between a retrieval system that works at 70% and one that works at 85%+.

The chunk size question is also underappreciated. Too small and you lose context. Too large and you dilute the relevant signal with noise. The right answer depends on your document structure and query distribution — and there's no substitute for actually measuring it.

The faithfulness problem

A RAG system that retrieves correctly can still hallucinate. The LLM will synthesise, extrapolate, and fill gaps — sometimes accurately, sometimes not. Without an explicit faithfulness check, you have no way to distinguish the two.

Faithfulness evaluation means checking whether every claim in the generated answer is grounded in the retrieved context. This is not easy to do at scale. But it's the difference between a system you can trust and one you're crossing your fingers about.

Good approaches include using a separate LLM call to verify each claim against the source passages, maintaining citation links so claims can be traced back to specific chunks, and flagging responses where the model hedges or qualifies heavily (a signal it's working from incomplete information).

What good evaluation looks like

The weakest form of RAG evaluation is manual spot-checking. You ask the system some questions you already know the answers to and see if it gets them right. This gives you a local minimum of confidence that tells you almost nothing about how the system behaves across the full query distribution.

A production-grade eval pipeline has:

• A ground truth dataset — ideally real questions from real users, with verified correct answers. If you're building from scratch, generate synthetic queries from your documents and manually verify a sample.
• Planted failures — deliberately bad test cases that your system should fail on (questions with no good answer in the corpus, ambiguous queries, multi-hop questions that require synthesising across documents). If your system passes these confidently, something is wrong.
• Quantitative metrics — retrieval recall@k, answer faithfulness rate, exact match on factual queries. Numbers you can track over time and compare across system versions.
• Regression coverage — any time the system fails on a real user query, that query goes into the eval set permanently. Your eval suite should grow with your failure cases.

Multi-hop queries are the hard part

Single-hop RAG (one question, one relevant passage) is relatively tractable. Multi-hop reasoning — where answering the question requires combining information from multiple documents — is where most RAG systems quietly break down.

The failure mode is subtle: the system retrieves documents that are individually relevant but doesn't correctly combine the information from them. The answer looks plausible but is wrong in a way that's hard to catch without knowing the correct answer.

Handling multi-hop queries well usually requires either a re-ranking step that looks at query-passage relevance in context of other retrieved passages, or an explicit decomposition step that breaks the question into sub-queries and assembles the answers.

The decisions that matter

When you're building a RAG pipeline, the decisions that matter most are:

• Why did you choose this chunk size and overlap strategy?
• Why hybrid retrieval over pure dense, or vice versa?
• How do you detect when retrieval has failed before sending to the LLM?
• What happens when the answer isn't in the corpus — does the system say so, or does it hallucinate confidently?
• How do you measure faithfulness, and what's your threshold for acceptable?

These are the questions a senior engineer would ask you in an interview. Being able to answer them with evidence — "we tried X, measured Y, and switched to Z because it improved recall by 12%" — is what demonstrates real competence.

A RAG pipeline that works is one you understand deeply enough to explain every decision, measure every failure mode, and fix the right thing when something breaks.

---

TryCrucible has RAG challenges at easy, medium, and hard difficulty — including a hard-mode challenge that tests multi-hop policy QA with a planted-failure eval dataset. See the RAG challenges →