What Is RAG in Generative AI? A Beginner-to-Advanced Guide

By Preeti Verma

Large language models are impressive, but they have two weaknesses that matter in real products. They do not know your private data, and they can state wrong things with confidence. Retrieval-augmented generation, usually shortened to RAG, is the most common way to deal with both.

This guide starts with the basic idea, shows a simple build, then goes into the parts that decide whether a RAG system works well: chunking, embeddings, hybrid search, reranking and evaluation. It ends with the mistakes that break real systems.

What RAG means in one paragraph

RAG means that before the model answers, your system retrieves relevant text from a knowledge source and puts it into the prompt. The model then answers using that text. The idea comes from a 2020 research paper, "Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks" by Patrick Lewis and colleagues, which combined a retriever with a generator and noted that the knowledge source can be swapped out to update what the system knows. Today the term is used more broadly for any pipeline of retrieve, then generate.

Why use RAG

  • Fresh and private knowledge. The model has not seen your company documents, your product manual or last week's policy change. Retrieval brings them in at question time.
  • Fewer made-up answers. If the answer is in the retrieved text, the model can quote it. You can also tell it to say "I do not know" when the text does not contain the answer.
  • Cheaper than retraining. Updating a document is far easier than fine-tuning a model.
  • Traceability. You can show the source passages that backed an answer, so users can verify.

RAG does not eliminate errors. If retrieval returns the wrong text, the model can still produce a wrong answer from it. That is why evaluation matters so much.

How a RAG pipeline works

A RAG system has two phases.

Phase 1: indexing (done ahead of time)

  1. Load documents: PDFs, web pages, wikis, tickets.
  2. Clean and split into chunks: smaller pieces of text.
  3. Create embeddings: each chunk is converted into a list of numbers (a vector) that represents its meaning.
  4. Store them in a vector database or an index, with metadata such as source, title and date.

Phase 2: answering (done for each question)

  1. Embed the question with the same embedding model.
  2. Retrieve the most similar chunks.
  3. Build a prompt containing the question and the retrieved chunks.
  4. Generate the answer from the model.
  5. Return the answer with sources.

A tiny example in Python

This is deliberately simple. It shows the idea without any specific vendor library. embed and ask_llm stand in for whichever embedding and chat model you use.

import numpy as np

def cosine(a, b):
    return float(np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b)))

# 1. Index
chunks = [
    "Refunds are available within 7 days of purchase for unused plans.",
    "Support is available by email from Monday to Saturday.",
    "A Pro subscription renews every month until it is cancelled.",
]
vectors = [embed(c) for c in chunks]

# 2. Retrieve
def retrieve(question, k=2):
    q = embed(question)
    scored = sorted(zip(chunks, vectors), key=lambda cv: cosine(q, cv[1]), reverse=True)
    return [c for c, _ in scored[:k]]

# 3. Generate
def answer(question):
    context = "\n".join(retrieve(question))
    prompt = (
        "Answer using only the context below. "
        "If the answer is not in the context, say you do not know.\n\n"
        f"Context:\n{context}\n\nQuestion: {question}"
    )
    return ask_llm(prompt)

Cosine similarity measures how close two vectors point, which is a common way to compare meaning. Real systems use a vector database or index so that search stays fast with millions of chunks.

The decisions that determine quality

Chunking

How you split text matters more than most beginners expect. Chunks that are too small lose context. Chunks that are too large dilute the match and waste the prompt. Practitioner guides commonly report that there is no universal best strategy, so you test on your own documents. Common approaches:

  • Fixed size with overlap: simple, and a sensible first version.
  • Recursive splitting: split by headings, paragraphs and sentences, falling back to size.
  • Structure-aware: respect sections, tables and code blocks.
  • Parent and child: retrieve a small chunk, then give the model its larger parent section for context.

Keep stable chunk IDs and store the source and position, so you can compare experiments and cite sources.

Embeddings

The embedding model decides what "similar" means. Pick one suited to your language and domain, use the same model for chunks and questions, and re-index if you change it. Test with your own questions, because a model that scores well on public benchmarks may not be best for your data.

Retrieval: dense, keyword and hybrid

Vector (dense) search finds text that is similar in meaning. Keyword search (such as BM25) finds exact words. Each fails on different questions: dense search can miss exact names, IDs and error codes, while keyword search can miss paraphrases. Many guides recommend hybrid search, which combines both and merges the rankings, often with a method called reciprocal rank fusion. It is one of the most consistent improvements reported.

Reranking

A first-stage retriever is fast but rough. A reranker, often a cross-encoder model, scores each candidate against the question more carefully. A common pattern is to retrieve around 50 candidates and rerank down to the best handful. It costs extra latency, so measure whether it helps on your data.

Prompting the generator

Tell the model to use only the supplied context, to quote or cite sources, and to say when the answer is missing. Put the context in clearly labelled sections. See prompt engineering for developers.

Access control

If different users may see different documents, enforce that at retrieval time. Filter chunks by the user's permissions before they reach the prompt. A model cannot keep a secret that you already gave it.

How to evaluate a RAG system

Evaluate retrieval and generation separately, so you know which part is failing.

Retrieval metrics check whether the right chunk was found:

  • Recall at k: is the correct chunk among the top k results?
  • MRR (mean reciprocal rank): how high does the first correct chunk rank?
  • nDCG: how good is the whole ranked list?

Generation metrics check the answer:

  • Faithfulness: does the answer stay within the retrieved text?
  • Answer relevance: does it actually answer the question?
  • Correctness: is it right, judged against a reference answer?

Build a test set of 50 to 100 real questions with expected answers and the source chunk that contains each. Run it after every change to chunking, embeddings, prompts or models. Tools such as RAGAS automate some of these checks, but spot-check manually too, because automated judges can be wrong.

The recommended order from practitioner guidance: start simple, measure, add hybrid search, measure again, add reranking, measure again, then tune chunk size against your own metrics.

RAG vs fine-tuning vs long context

  • RAG: best when knowledge changes, is private or large, and you need sources.
  • Fine-tuning: best for changing style, format or a narrow behaviour, not for adding lots of facts.
  • Long context: putting entire documents into the prompt works for small knowledge sets, but cost and latency grow, and models can still miss details in very long inputs.

They can be combined. Start with the simplest option that meets the need.

Common mistakes

  1. No evaluation. Without a test set you cannot tell if a change helped.
  2. Chunks that split the answer across pieces. Use overlap or parent chunks.
  3. Ignoring metadata. Dates, versions and document types help filter stale or irrelevant content.
  4. Trusting dense search alone for names, IDs and codes.
  5. Stuffing too many chunks into the prompt. More text is not always better, and the model may ignore the middle.
  6. Forgetting prompt injection. A retrieved document can contain instructions aimed at the model. Treat retrieved text as untrusted data, and never let it trigger actions without checks.
  7. Stale indexes. Re-index when documents change.
  8. Skipping access control. Retrieval must respect permissions.

Real-world applications

  • A support assistant that answers from help articles and past tickets.
  • Internal search across policies, wikis and runbooks.
  • Legal and compliance question answering with citations.
  • A coding assistant that retrieves from your own repository.
  • A study tool that answers from course notes.

Build it yourself: a learning project

  1. Choose 20 to 50 documents you understand well.
  2. Split, embed and index them.
  3. Write 30 test questions with expected answers.
  4. Build the pipeline and score it.
  5. Try one improvement at a time (chunk size, hybrid search, reranking) and record the score.
  6. Write a short README with the results.

This project is a strong addition to an AI-focused resume. See our AI skills guide and top projects for students. If you can explain RAG clearly out loud, you are ready for the interview question. Practise with the GenQuest AI mock interview.

A worked evaluation example

Evaluation sounds abstract until you do it once. Suppose you have a small help-centre knowledge base and you write five test questions. For each one you note which chunk contains the answer.

Run retrieval with k = 3 (the top three chunks) and record the position of the correct chunk, or "not found":

  • Question 1: correct chunk ranked 1.
  • Question 2: correct chunk ranked 2.
  • Question 3: correct chunk not in the top 3.
  • Question 4: correct chunk ranked 1.
  • Question 5: correct chunk ranked 3.

Recall at 3 is the share of questions whose correct chunk appears in the top three: four out of five, so 0.8. MRR averages the reciprocal of each rank (1, 1/2, 0, 1, 1/3), which is (1 + 0.5 + 0 + 1 + 0.33) / 5, about 0.57.

Now you can act. Question 3 failed, so open it. Maybe the answer was split across two chunks (use overlap or parent chunks), or the question used different words from the document (try hybrid search). Change one thing, re-run the five questions and see whether the numbers move. That loop, with a bigger test set, is how real teams improve RAG.

Frequently asked questions

Do I need a vector database to build RAG?

For a small project, no. A list of vectors in memory works. For larger data you need an index or database built for similarity search. Some general-purpose databases also support vectors through extensions.

How many chunks should I retrieve?

Start with three to five, and tune using your evaluation set. Retrieving more can raise recall but also adds noise and cost.

Why does my RAG system still give wrong answers?

Check retrieval first: was the right chunk retrieved? If yes, the problem is the prompt or the model. If no, improve chunking, embeddings or add hybrid search.

Is RAG dead because of long context windows?

No. Long context helps for small sets of documents, but RAG still wins on cost, latency, large or changing knowledge, access control and citations.

Sources

The practitioner guides above are vendor and community articles. Their specific benchmark numbers are unverified, so use them for the shape of the approach and test on your own data.

The takeaway

RAG is retrieve, then generate. Start with a simple pipeline, build a test set, and improve one piece at a time: chunking, hybrid search, reranking and the prompt. Measure retrieval and answers separately, protect access and treat retrieved text as untrusted. A small system you can measure beats a clever one you cannot.