Building a RAG Data Pipeline: Ingestion, Chunking, and Embedding That Actually Scale

RAG systems fail at retrieval far more often than they fail at generation. The language model, given good context, is usually capable of producing a correct answer. The hard problem is getting the right context into the prompt in the first place — and that's determined entirely in the indexing phase.

The data pipeline you build before a user sends their first query decides how good your retrieval will ever be. This guide covers the decisions that matter most: ingestion architecture, chunking strategy, embedding choice, and the metadata that makes re-ranking work.

What the RAG data pipeline actually is

RAG stands for Retrieval-Augmented Generation. The "retrieval" part requires two pipelines running at different times:

• The indexing pipeline (offline, runs once or on schedule): loads documents, processes them into chunks, generates embeddings, stores everything in a vector database
• The query pipeline (online, runs per user request): embeds the query, searches the index, retrieves top-k chunks, augments the prompt

Most engineers focus on the query pipeline because it's what the user sees. The indexing pipeline is where retrieval quality is actually determined.

Stage 1 — Document ingestion

Ingestion is about getting raw source data into a form your pipeline can process. The decisions here are mostly practical, but a few have real quality implications.

Source types and parsers. PDFs are the most common source, and the most problematic. A text-layer PDF is straightforward. A scanned PDF needs OCR (Tesseract, AWS Textract, or a vision model). A PDF with tables and diagrams loses structure when you extract text — whether that matters depends on whether your queries target that structure.

For each source type, use a parser that preserves the structure you'll need at query time. If headers, sections, and document hierarchy matter for retrieval, parse them explicitly rather than treating the document as a flat string.

Handling updates. A pipeline that ingests once is a prototype. A production pipeline handles document updates: detecting what changed, re-ingesting changed documents, removing deleted documents from the index. Build this from the start or you'll spend weeks retrofitting it later.

The simplest approach: store a hash of each document alongside its chunks. On each pipeline run, compare hashes. Re-embed only what changed. This keeps incremental costs low as your corpus grows.

Stage 2 — Chunking strategy

Chunking is where most data pipelines make their biggest mistakes. The chunk is the unit of retrieval — it's what gets returned when a query matches, and it's what goes into the prompt. If your chunks are too small, they lack context. If they're too large, they dilute the relevant signal with noise.

Fixed-size chunking (e.g., 512 tokens, 100-token overlap) is the default in most tutorials because it's easy to implement. It's also naive: it cuts sentences and paragraphs mid-thought without regard for meaning. Use it as a baseline, not as a production strategy.

Semantic chunking groups text by meaning rather than by token count. Sentences that are semantically related stay together; topic boundaries become chunk boundaries. This requires more processing but produces chunks that are far more coherent as retrieval units.

Document-structure chunking uses the document's own hierarchy — headers, sections, paragraphs — to define chunk boundaries. This is often the best strategy when documents have clear structure. A technical specification, a policy document, a product manual — these have meaningful sections that are natural retrieval units.

The overlap question. Overlapping chunks (where the end of chunk N is repeated at the start of chunk N+1) reduce the risk of splitting a relevant passage across two chunks. A 10–20% overlap is common. More than that and you're duplicating retrieved context in your prompts without much benefit.

The only way to know what chunk size is right for your use case is to measure it. Build a small eval set of 20–30 representative queries with known correct answers, run retrieval at different chunk sizes, and measure recall@k (how often is the relevant chunk in the top-k results?). This measurement will tell you more than any tutorial recommendation.

Stage 3 — Embedding

The embedding model converts text into a dense vector. Two pieces of text with similar meanings get similar vectors. The search step finds chunks whose vectors are closest to the query vector.

Choosing an embedding model. For most use cases, text-embedding-3-small (OpenAI) or text-embedding-3-large is a strong default. small is faster and cheaper; large is more accurate. For multilingual content, models like multilingual-e5-large handle non-English text better than OpenAI's models.

Benchmark embedding models on your actual data before committing. The MTEB leaderboard (Massive Text Embedding Benchmark) gives general rankings, but performance varies significantly by domain and query type. A model that ranks highly on MTEB may not be the best choice for your specific corpus.

Embedding documents vs queries differently. Some embedding models are trained with an asymmetric structure — one embedding space for long documents, a different (or the same) space for short queries. Models like E5 family use this pattern: you prefix documents with "passage: " and queries with "query: " before embedding. Ignoring this distinction degrades retrieval quality significantly for these models.

Batch embedding costs and latency. Re-embedding a large corpus after switching models is expensive. OpenAI's embedding API charges per token; at 100,000 documents with 500 tokens each, that's 50 million tokens. Calculate this before choosing a model and before each re-indexing run.

Stage 4 — Storing with metadata

Vector databases store vectors and return the chunks whose vectors are closest to a query vector. But what you store alongside the vectors — the metadata — is what makes filtering, re-ranking, and attribution possible.

Essential metadata to store per chunk:

• Source document identifier (file name, URL, database ID)
• Position in source (section, page number, paragraph index)
• Document creation/modification date
• Any access control tags (if different users should see different documents)

Why metadata matters for retrieval quality. With metadata, you can filter the vector search to a subset of documents before running similarity search. A query about "Q3 pricing policy" can be restricted to documents tagged as pricing-related, reducing noise in the top-k results. Without metadata, every chunk in the entire index competes for retrieval on every query.

Hybrid retrieval. The most reliable production RAG systems combine dense retrieval (vector similarity) with sparse retrieval (BM25 keyword matching) and re-rank the combined results. Dense retrieval captures semantic similarity; sparse retrieval captures exact keyword matches that embeddings sometimes miss (product codes, names, specific numbers). Implementing hybrid retrieval consistently improves recall at the cost of some added complexity.

The pipeline in code

from openai import OpenAI
import hashlib

client = OpenAI()

def embed_chunks(chunks: list[str]) -> list[list[float]]:
    response = client.embeddings.create(
        model="text-embedding-3-small",
        input=chunks,
    )
    return [item.embedding for item in response.data]

def build_index(documents: list[dict], vector_store) -> None:
    for doc in documents:
        doc_hash = hashlib.md5(doc["content"].encode()).hexdigest()

        # Skip if content unchanged
        if vector_store.hash_exists(doc["id"], doc_hash):
            continue

        chunks = chunk_document(doc["content"])
        embeddings = embed_chunks(chunks)

        records = [
            {
                "id": f"{doc['id']}_chunk_{i}",
                "embedding": embedding,
                "text": chunk,
                "metadata": {
                    "source_id": doc["id"],
                    "source_title": doc.get("title", ""),
                    "chunk_index": i,
                    "doc_hash": doc_hash,
                },
            }
            for i, (chunk, embedding) in enumerate(zip(chunks, embeddings))
        ]

        vector_store.upsert(records)

Measuring whether it works

A data pipeline is only as good as the retrieval it enables. Measure recall@k on a representative eval set before and after any pipeline change. If you change chunk size, embedding model, or overlap strategy, run the eval set again. Don't trust intuition — the relationship between pipeline choices and retrieval quality is not always obvious.

The eval set should cover:

• Questions with clear single-passage answers
• Questions requiring synthesis across multiple chunks
• Exact-match queries (specific names, numbers, codes)
• Semantic queries (conceptual questions with no exact match in the document)

A retrieval system that scores 90%+ on recall@5 for a representative eval set is genuinely production-ready. One that hasn't been measured against an eval set is not, regardless of how good it looks in the demo.

---

TryCrucible's RAG pipeline challenge evaluates your full implementation — ingestion, retrieval, and faithfulness — against hidden test inputs and a scoring rubric that covers correctness, architecture, and robustness. Completing it gives you a scored artifact on your public profile. Try the RAG challenge →