Building RAG (Retrieval-Augmented Generation) Pipelines

Introduction

RAG has come up implicitly in nearly every lesson since lesson 10 — LlamaIndex's query engine, a vector database's .query() call, and embeddings all exist to support it. This lesson assembles all three pieces into one complete, explicit pipeline built from raw calls, so the pattern is fully visible end to end.

A Real-Life Analogy First

Think about the difference between a closed-book exam and an open-book exam. In a closed-book exam, a student answers purely from what they memorized beforehand — if they never studied a fact, they cannot know it, no matter how smart they are. In an open-book exam, the same student can flip to the exact right page of a textbook before answering, so even a fact they never memorized can be answered correctly and precisely, as long as it is somewhere in the book they are allowed to consult.

The Four Steps of RAG

A Complete RAG Pipeline in Action

Use case: this example builds all four steps explicitly with Chroma and the OpenAI SDK — the same pattern LlamaIndex's query_engine.query() automated behind the scenes in lesson 10.

import chromadb
from openai import OpenAI

client_openai = OpenAI()
client_chroma = chromadb.PersistentClient(path="./chroma-db")
collection = client_chroma.get_or_create_collection("support-docs")

# Step 1 & 2: Index and store
collection.add(
    ids=["doc-1", "doc-2"],
    documents=[
        "Refunds are processed within 5-7 business days of approval.",
        "Shipping takes 3-5 business days for domestic orders.",
    ],
)

def answer_question(question: str) -> str:
    # Step 3: Retrieve the most relevant chunk(s)
    results = collection.query(query_texts=[question], n_results=2)
    context = "\n".join(results["documents"][0])

    # Step 4: Generate a grounded answer
    response = client_openai.chat.completions.create(
        model="gpt-4o-mini",
        messages=[
            {"role": "system", "content": "Answer only using the provided context. If the answer isn't in the context, say you don't know."},
            {"role": "user", "content": f"Context:\n{context}\n\nQuestion: {question}"},
        ],
    )
    return response.choices[0].message.content

print(answer_question("How long until I get my refund?"))

Chunking Strategy

The two example documents above happened to be short enough to embed whole. Real documents need to be split into smaller chunks first (a few hundred words each, often with some overlap between consecutive chunks) — too large a chunk dilutes its embedding's specificity, while too small a chunk loses surrounding context the model would need to answer well. This is similar to how a well-organized textbook is split into digestible sections rather than one giant unbroken wall of text — a section that is too broad forces you to read too much to find the answer, while a section that is too narrow might cut off half of the explanation you actually needed.

RAG vs Fine-Tuning

A common question is whether to use RAG or fine-tune a model (lesson 17) to "teach" it your data. RAG is almost always the right starting point: it is faster to set up, keeps data current without retraining, and lets you cite exactly which source grounded an answer — fine-tuning is better reserved for teaching a model a specific style, format, or narrow skill rather than injecting knowledge. Continuing the exam analogy: RAG is giving a student an open book; fine-tuning is more like giving a student months of extra practice tests on one narrow topic so their instincts improve — the two solve genuinely different problems.

Common Mistakes

Best Practices

Frequently Asked Questions

Key Takeaways

Summary

RAG is the single most common pattern in real generative AI applications, and now that you have built one from scratch, every framework's higher-level RAG abstraction (LlamaIndex's query engine, LangChain's retrievers) will make a lot more sense.