InterviewKata
AI engineering · Module 15 of 21 · Level 4: Build AI features

Embeddings and RAG: answering from your own documents

Models don't know your handbook, your tickets or your product docs, and those are too big to paste into every request. Retrieval-augmented generation (RAG) finds the few passages that matter and gives only those to the model. It's Module 03's "open book", automated, and the most common pattern in AI products.

1. The idea

A good librarian doesn't hand you the whole library. You ask a question; they think about what it means, pull the five most relevant pages from different shelves, and you answer from those pages, pointing at the page number for each claim.

Two ideas make it work:

  1. Embeddings turn meaning into numbers. An embedding model turns a piece of text into a list of numbers (a vector, often 1,024 or more of them) so that texts with similar meaning get similar numbers. "How do I get my money back?" and "refund policy" end up close together even though they share no words.
  2. RAG retrieves, then generates. Ahead of time you split documents into chunks, embed them and store them (the index). At question time you embed the question, find the closest chunks (plus a keyword search), put the best ones into the prompt, and ask the model to answer only from them, with citations.

Similarity is usually measured by the angle between vectors (cosine similarity): 1 means the same direction (same meaning), near 0 means unrelated.

2. How it works

Indexing happens once (and whenever documents change). Retrieval and generation happen on every question.

3. Try it yourself: the meaning map

Real embeddings have hundreds of dimensions; this map squeezes some help-centre chunks into two so you can see the idea. Pick a question: the closest chunks (by meaning) are retrieved. Compare with keyword matching on the right.

Top 3 by meaning

    Keyword match only

      Gold dot = your question; green = the 3 retrieved chunks. Illustrative 2-D positions, hand-placed to show the idea. Real systems compare full vectors from an embedding model.

      4. Why it matters at work

      Most RAG quality problems are retrieval problems: the right passage never reached the model. What actually moves the needle:

      1. Chunk sensibly. Split by headings and paragraphs, a few hundred tokens each, with a little overlap. Too small loses context; too big dilutes the match.
      2. Add context to each chunk. "Revenue grew 3%" means nothing alone. Anthropic's contextual retrieval prepends a one-line summary of where each chunk sits in its document before indexing it.
      3. Combine meaning and keywords (hybrid search). Embeddings catch paraphrases; keyword search (BM25) catches exact names, codes and error messages. Use both.
      4. Rerank. Retrieve broadly (say 50 chunks), then let a reranker model pick the best 5–20 for the prompt.
      5. Answer only from the sources, with citations, and allow "not found" (Module 03).
      6. Measure retrieval separately from answers: did the right chunk appear in the top results? (Module 17)

      Anthropic measured the effect of these techniques on retrieval failures (how often the right chunk was missing from the top 20):

      SetupFailure rate
      Standard embeddings5.7%
      Contextual embeddings3.7% (−35%)
      Contextual embeddings + contextual BM252.9% (−49%)
      …plus reranking1.9% (−67%)
      When you don't need RAGIf your documents fit comfortably in the context window and you can afford it, just include them (and cache them, Module 04). Coding agents mostly skip vector search too: they search code with exact tools like grep, which suits code well. Reach for RAG when the material is large, changing, or needs access control per user.

      5. Embedding models and built-in options

      Checked against vendor documentation, October 2026.

      ProviderEmbedding modelsNotes
      Voyage AI (MongoDB)voyage-4-large, voyage-4, voyage-4-lite, voyage-code-4; reranker rerank-3Anthropic has no embedding model of its own and points to Voyage. 32K-token inputs; contextualized and multimodal variants
      Googlegemini-embedding-2Multimodal (text, images, audio, video, PDF); 8,192 tokens; 128–3,072 dimensions
      OpenAItext-embedding-3-small, text-embedding-3-large8,192 tokens; 1,536 and 3,072 dimensions, reducible

      Built-in retrieval: instead of building the whole pipeline, you can use a vendor's managed option: OpenAI's file search tool in the Responses API, Google's File Search tool in the Gemini API, or Claude's citations for documents you pass in. In editors, Cursor and Copilot index your codebase with embeddings for their semantic search.

      Storage: any vector database works (pgvector in Postgres, MongoDB, Elasticsearch, or a dedicated one). Start with what your stack already has.

      6. Remember it

      Memory hook: "Index, fetch, cite"Prepare the library once, fetch the few right pages for each question, and make every answer point to its page.
      How this hook works: what “Index, fetch, cite” means

      What the slogan means, phrase by phrase

      “Index”
      Ahead of time: chunk the documents, add context to each chunk, embed them, and also build a keyword index. Re-index when documents change.
      “fetch”
      Per question: embed it, search by meaning and by keywords, rerank, and keep the best few chunks.
      “cite”
      Generate the answer only from those chunks, citing each one, and say "not found" when they don't cover it.

      Why this picture fits: each word is a stage you can test and fix on its own. Bad answers usually trace back to one stage: a missing document (index), the wrong passages (fetch), or invented claims (cite).

      The story → the concept

      The librarianRAG
      Cataloguing books by topicChunking + embeddings + keyword index
      A note on each page saying which book it's fromContextual retrieval
      Understanding what you mean, not just your wordsVector (semantic) search
      Also searching for the exact part numberKeyword search (BM25), hybrid
      Picking the best five of fifty pagesReranking
      "Page 12 says…"Citations
      "The library doesn't have that"Answering "not found"

      Watch it on an example

      1. Index: your 400-page help centre becomes 3,000 chunks, each prefixed with "From: Returns policy › Damaged items".
      2. Fetch: "my parcel arrived crushed, what now?" matches the damaged-items chunks by meaning (no shared words), and the keyword search adds the chunk mentioning "crushed".
      3. Cite: "You can request a replacement within 30 days [Returns policy, §4]." Every claim is checkable.

      The trap it saves you from

      Blaming the model for wrong answers when the right passage never reached it. Check "fetch" first.

      It still works when…

      An agent does the retrieving (Module 16): the search becomes a tool it can call several times, refining the query, which is called agentic retrieval.

      7. Common mistakes

      Retrieving what the user isn't allowed to seeFilter by permissions during retrieval. Once a confidential chunk is in the prompt, the model can repeat it.
      Embeddings onlyPure semantic search misses exact identifiers ("error E-4031", product codes). Add keyword search.
      Chunks without context"It increased by 3%" retrieves badly and answers worse. Add document context to chunks.
      Mixing embedding modelsQuery and documents must be embedded with the same model (and the right query/document setting). Changing models means re-indexing everything.
      No "not found" pathIf retrieval returns nothing useful, the model should say so, not answer from memory.

      8. Hands-on lab (roadmap weeks 16–17)

      1. Pick a document set you know well (your project docs, a handbook) and write 20 questions with the passage that answers each.
      2. Build the simplest RAG: chunk, embed with one provider, store, retrieve the top 5, answer with citations. Your coding agent can scaffold it.
      3. Measure retrieval: for how many of the 20 questions is the right passage in the top 5?
      4. Improve one stage at a time: add keyword search, then chunk context, then a reranker. Re-measure after each.
      5. Test "not found" with three questions your documents can't answer.

      9. Self-check quiz

      What does an embedding capture?

      That's why "refund policy" can match "how do I get my money back?" with no shared words.

      Questions quoting exact error codes retrieve the wrong chunks. Fix?

      Hybrid search: embeddings for meaning, keywords for exact identifiers.

      RAG gave a wrong answer. What do you check first?

      Most failures are retrieval failures: if the passage never reached the model, no prompt can fix it.

      Where must per-user access control happen?

      Anything in the prompt can be repeated. Filter at retrieval time.

      10. Go deeper

      ResourceTimeWhy
      Introducing contextual retrieval (Anthropic)20 minThe chunk-context technique and the measurements in section 4.
      Embeddings (Claude docs)15 minVoyage models, query vs document embeddings, quantisation.
      Embeddings (Gemini) and Embeddings (OpenAI)20 minThe other two providers' models and options.
      Mastering LLMs open course, RAG talks2 hPractitioners on what actually improves RAG in production.