Ask-Your-Notes Q&A Bot (RAG)

Combine Chroma retrieval with Claude to answer questions about your own notes, with citations and an "I don't know" safeguard.

RAGPrompt DesignAnthropic SDKCitations

Overview

A language model knows a lot about the world but nothing about your meeting notes, your company wiki or last week's decisions. Retrieval-Augmented Generation (RAG) fixes that: first retrieve the most relevant pieces of your own documents, then give them to the model together with the question and ask it to answer from those pieces only.

This project builds a RAG bot on top of the Semantic Notes Search project. For each question it retrieves the closest note chunks from Chroma, skips weak matches, wraps the chunks in clearly labelled tags, and asks Claude to answer using only those notes, cite the file each fact came from, and say so plainly when the notes do not contain the answer.

What You'll Build

A retrieve() function that reuses the Chroma collection and drops chunks above a distance threshold.
A prompt that places retrieved notes inside <note source="..."> tags.
A system prompt with grounding rules: answer only from the notes, cite sources, admit when the answer is missing.
A streamed answer with client.messages.stream(), plus a list of the sources used.
An interactive question loop that skips the API call entirely when nothing relevant is found.

Prerequisites

The Semantic Notes Search project, with your notes already indexed.
An Anthropic API key in ANTHROPIC_API_KEY — lesson 3.
Building RAG pipelines — lesson 15.
Prompt engineering basics — lesson 16.

Project Structure

notes-search/
├── notes_search.py      # from the previous project (provides get_collection)
├── ask_notes.py         # NEW: the Q&A bot
├── notes/
└── notes-db/

The bot lives in the same folder as notes_search.py and imports its get_collection() function, so both tools always read the same database. The pipeline has three clear stages — retrieve, build prompt, generate — each in its own function, which makes it easy to test retrieval quality on its own before spending any API tokens.

pip install anthropic   # chromadb is already installed from the previous project

Step 1: Retrieve Relevant Chunks

Asking for the top four chunks always returns four results, even when none of them is relevant. The distance threshold removes weak matches so they cannot mislead the model. The right threshold depends on your embedding model; print the distances for a few good and bad questions and pick a value between them.

from notes_search import get_collection

MAX_DISTANCE = 1.5  # tune by inspecting distances for good and bad questions


def retrieve(question: str, k: int = 4) -> list[dict]:
    collection = get_collection()
    if collection.count() == 0:
        return []
    results = collection.query(query_texts=[question], n_results=min(k, collection.count()))
    chunks = []
    for doc, meta, distance in zip(
        results["documents"][0], results["metadatas"][0], results["distances"][0]
    ):
        if distance <= MAX_DISTANCE:
            chunks.append({"source": meta["source"], "text": doc, "distance": distance})
    return chunks

Step 2: Build a Grounded Prompt

Tags such as <note source="team-meeting.md"> make the boundary between your data and the question unambiguous, and give the model the exact file names to cite. The rules go in the system prompt so they apply to every question.

SYSTEM_PROMPT = """You answer questions about the user's personal notes.
Use ONLY the information inside the <notes> tags.
- If the notes do not contain the answer, reply exactly: I couldn't find that in your notes.
- After each fact, cite the file it came from in square brackets, e.g. [git-cheatsheet.md].
- Keep answers short and practical."""


def build_prompt(question: str, chunks: list[dict]) -> str:
    notes = "\n".join(
        f'<note source="{chunk["source"]}">\n{chunk["text"]}\n</note>' for chunk in chunks
    )
    return f"<notes>\n{notes}\n</notes>\n\nQuestion: {question}"

Step 3: Stream the Answer

def answer(question: str) -> None:
    chunks = retrieve(question)
    if not chunks:
        print("No relevant notes found - skipping the API call.")
        return

    print("Sources: " + ", ".join(f"{c['source']} ({c['distance']:.2f})" for c in chunks))
    with client.messages.stream(
        model=MODEL,
        max_tokens=64000,
        system=SYSTEM_PROMPT,
        messages=[{"role": "user", "content": build_prompt(question, chunks)}],
    ) as stream:
        for text in stream.text_stream:
            print(text, end="", flush=True)
        final = stream.get_final_message()
    print()
    if final.stop_reason == "refusal":
        print("[Claude declined to answer this question.]")

Step 4: An Interactive Loop

Each question is answered independently — there is no chat history — which keeps every answer grounded in freshly retrieved notes and keeps the cost per question predictable. The same error-handling chain from the chatbot project keeps the loop running through network hiccups.

Complete Code

ask_notes.py

"""ask_notes.py - answer questions about your notes with retrieval + Claude.

First index your notes:  python notes_search.py index ./notes
Then run:                python ask_notes.py
"""
import anthropic

from notes_search import get_collection

MODEL = "claude-opus-5-5"
MAX_DISTANCE = 1.5  # tune by inspecting distances for good and bad questions

SYSTEM_PROMPT = """You answer questions about the user's personal notes.
Use ONLY the information inside the <notes> tags.
- If the notes do not contain the answer, reply exactly: I couldn't find that in your notes.
- After each fact, cite the file it came from in square brackets, e.g. [git-cheatsheet.md].
- Keep answers short and practical."""

client = anthropic.Anthropic()  # reads ANTHROPIC_API_KEY from the environment


def retrieve(question: str, k: int = 4) -> list[dict]:
    collection = get_collection()
    if collection.count() == 0:
        return []
    results = collection.query(query_texts=[question], n_results=min(k, collection.count()))
    chunks = []
    for doc, meta, distance in zip(
        results["documents"][0], results["metadatas"][0], results["distances"][0]
    ):
        if distance <= MAX_DISTANCE:
            chunks.append({"source": meta["source"], "text": doc, "distance": distance})
    return chunks


def build_prompt(question: str, chunks: list[dict]) -> str:
    notes = "\n".join(
        f'<note source="{chunk["source"]}">\n{chunk["text"]}\n</note>' for chunk in chunks
    )
    return f"<notes>\n{notes}\n</notes>\n\nQuestion: {question}"


def answer(question: str) -> None:
    chunks = retrieve(question)
    if not chunks:
        print("No relevant notes found - skipping the API call.")
        return

    print("Sources: " + ", ".join(f"{c['source']} ({c['distance']:.2f})" for c in chunks))
    with client.messages.stream(
        model=MODEL,
        max_tokens=64000,
        system=SYSTEM_PROMPT,
        messages=[{"role": "user", "content": build_prompt(question, chunks)}],
    ) as stream:
        for text in stream.text_stream:
            print(text, end="", flush=True)
        final = stream.get_final_message()
    print()
    if final.stop_reason == "refusal":
        print("[Claude declined to answer this question.]")
    print(f"[tokens - in: {final.usage.input_tokens}, out: {final.usage.output_tokens}]")


def main() -> None:
    print("Ask a question about your notes (or press Enter to quit).")
    while True:
        try:
            question = input("\nQuestion: ").strip()
        except (EOFError, KeyboardInterrupt):
            break
        if not question:
            break
        try:
            answer(question)
        except anthropic.APIConnectionError:
            print("[Network problem - check your connection and try again.]")
        except anthropic.RateLimitError:
            print("[Rate limited - wait a moment and try again.]")
        except anthropic.APIStatusError as err:
            print(f"[API error {err.status_code}: {err.message}]")
    print("Goodbye!")


if __name__ == "__main__":
    main()

Sample Run

python ask_notes.py (example session - the Sources lines are from a real run; Claude's wording and token counts will vary)

Click Run to see what this code prints.

The third question shows why the distance threshold matters: nothing in the notes is about geography, so no chunks pass the filter and the bot does not spend tokens or invent an answer. If a weak chunk does slip through, the system prompt's rule makes Claude reply "I couldn't find that in your notes." instead of answering from general knowledge.

Extend This Project

Use the API's built-in citations feature by sending each chunk as a document content block with citations: {"enabled": true}, and print the exact quoted text for each claim.
Add prompt caching for a large, fixed set of notes so repeated questions cost less.
Build a small evaluation set of 10 questions with known answers and measure how often the bot is right before and after changing k or MAX_DISTANCE.
Add conversation memory (from the chatbot project) so follow-up questions like "who owns that?" work.
Wrap the bot in a simple web UI with Flask or FastAPI.

Summary

You built a complete RAG pipeline: retrieval from a vector database, filtering weak matches, a grounded prompt with clearly tagged sources, and a model instructed to cite its sources and admit when it does not know. These are the core ideas behind company knowledge-base assistants, documentation bots and "chat with your PDF" tools.