Skip to content
AI.info

Implementation Guides

Building a RAG System from Scratch

A complete step-by-step guide to building a production-ready Retrieval-Augmented Generation system. Covers document processing, chunking, embeddings, vector stores, hybrid retrieval with Cohere Rerank 4, cited generation, and evaluation.

Building a RAG System from Scratch

Gabriele Masetti ·

Architecture

A production RAG system has five moving parts: an ingestion pipeline that turns raw documents into searchable chunks, an embedding model that maps text to vectors, a vector store that indexes those vectors for approximate nearest-neighbor (ANN) search, a retrieval-plus-rerank layer that narrows candidates down to the few passages worth showing the model, and a generation step that assembles a prompt and produces a cited answer. Evaluation sits alongside all of it, not after it — you need a way to measure retrieval quality independently from generation quality, because a system can fail at either stage for unrelated reasons.

The data flow is:

documents -> loaders -> chunker -> embedder -> vector store (index)
                                                      |
query -> embedder -> ANN search (top-k) -> reranker -> top-n -> prompt -> LLM -> answer + citations

Treat ingestion and query-time retrieval as separate services from day one. Ingestion is a batch/streaming job you run on a schedule or on webhook triggers when source documents change; retrieval is a low-latency read path. Conflating them is the single biggest reason RAG prototypes don't survive contact with production traffic.

Ingestion and Chunking

Start with document loading. For PDFs, pypdf or the unstructured library (which also handles DOCX, HTML, PPTX, and scanned images with OCR fallback) give you clean text plus per-page/per-element metadata. Preserve source metadata (document ID, page number, section heading, URL, last-modified date) at load time — you cannot cheaply recover it later, and you need it for citations and for filtered retrieval (e.g., "only search docs updated after X").

Chunking is the highest-leverage decision in the whole pipeline. Two failure modes dominate: chunks too large dilute the embedding (the vector represents an average of several ideas, so semantic search gets fuzzy), and chunks too small lose the surrounding context the LLM needs to answer correctly. RecursiveCharacterTextSplitter from langchain_text_splitters is a reasonable default — it splits on paragraph breaks first, then sentences, then words, only falling back to hard character cuts when a paragraph itself exceeds chunk_size:

from langchain_text_splitters import RecursiveCharacterTextSplitter

splitter = RecursiveCharacterTextSplitter(
    chunk_size=800,       # characters, not tokens
    chunk_overlap=120,    # preserves context across chunk boundaries
    separators=["\n\n", "\n", ". ", " ", ""],
)

chunks = splitter.split_text(document_text)

For technical or structured content (API docs, legal contracts, code), prefer structure-aware splitting: split on Markdown headers or HTML tags first, then recursively split any section that's still too large. That keeps a chunk's retrieved context aligned with a coherent unit (one function, one clause, one FAQ entry) rather than an arbitrary character window. A chunk size of 500–1,000 characters (roughly 150–300 tokens) with 10–20% overlap is a solid starting point for prose; tune it against your own eval set rather than trusting a blog-post default.

Attach stable, deterministic chunk IDs (hash of source ID + offset) so re-ingesting an unchanged document doesn't create duplicate vectors, and so you can delete/replace exactly the chunks belonging to an updated source document.

Embeddings

OpenAI's text-embedding-3-small (1,536 dimensions, $0.02 per 1M input tokens) and text-embedding-3-large (3,072 dimensions, $0.13 per 1M input tokens) are the standard hosted option; both support the Matryoshka representation trick — pass a dimensions parameter to get a shorter, still-useful vector (e.g., 256 or 512) and cut storage/search cost with a small recall hit.

Those are list prices for synchronous calls; routing ingestion through the Batch API halves them, which is worth doing for the first full index of a corpus — the largest single embedding bill you will pay.

If you need to keep embeddings on your own infrastructure or avoid per-call cost, open models from the sentence-transformers ecosystem — BAAI/bge-small-en-v1.5 or sentence-transformers/all-MiniLM-L6-v2 — run locally on CPU or a small GPU and are competitive on English retrieval benchmarks.

Component Spec
text-embedding-3-small 1,536 dims, $0.02 per 1M input tokens
text-embedding-3-large 3,072 dims, $0.13 per 1M input tokens
Cohere rerank-v4.0-pro / -fast 32K-token context, multilingual 
from openai import OpenAI

client = OpenAI()

def embed(texts: list[str], model: str = "text-embedding-3-small") -> list[list[float]]:
    resp = client.embeddings.create(model=model, input=texts)
    return [d.embedding for d in resp.data]

Batch embedding calls — send 100–500 chunks per request rather than one at a time, both models accept batched input lists — and cache the results keyed by chunk ID so re-runs of the ingestion job skip unchanged chunks. Whatever model you pick for ingestion, you must use the exact same model (and dimensions setting) for queries at retrieval time; embeddings from different models are not comparable.

Vector Store

For most teams already running Postgres, pgvector is the pragmatic choice — one fewer system to operate, transactional consistency with your existing metadata tables, and it scales to tens of millions of vectors with an HNSW index:

CREATE TABLE chunks (
    id UUID PRIMARY KEY,
    document_id UUID NOT NULL,
    content TEXT NOT NULL,
    metadata JSONB,
    embedding VECTOR(1536)
);

CREATE INDEX ON chunks USING hnsw (embedding vector_cosine_ops)
    WITH (m = 16, ef_construction = 64);

HNSW builds incrementally (no separate training pass, unlike IVFFlat) and gives the best recall/latency tradeoff for read-heavy workloads, at the cost of higher memory and slower inserts — fine for most RAG corpora, which are read-dominated. If you outgrow a single Postgres instance or need a fully managed, horizontally-scaled ANN service, purpose-built stores (Chroma for local/embedded use, or a hosted vector database) are worth the migration.

Chroma is the fastest path to a working local prototype:

import chromadb

client = chromadb.PersistentClient(path="./chroma_db")
collection = client.get_or_create_collection("docs")

collection.add(ids=chunk_ids, embeddings=chunk_vectors,
                documents=chunk_texts, metadatas=chunk_metadata)

results = collection.query(query_embeddings=[query_vector], n_results=20)

FAISS (faiss-cpu/faiss-gpu) is the library to reach for when you want raw ANN performance and are willing to manage persistence and metadata storage yourself — it's a search index, not a database:

import faiss

index = faiss.IndexHNSWFlat(1536, 32)   # d=1536, M=32
index.hnsw.efConstruction = 64
index.hnsw.efSearch = 64
index.add(vectors)                       # normalize vectors first for cosine similarity
distances, ids = index.search(query_vector.reshape(1, -1), k=20)

Whichever store you pick, always retrieve metadata filters alongside the vector search (document type, date range, access control tags) — pure semantic search with no filtering is how RAG systems leak content across tenants or surface stale documents ranked above current ones.

Retrieval and Reranking

Retrieve more than you plan to use. Pull the top 15–30 candidates by vector similarity (or hybrid: vector + BM25/keyword search, especially important for exact terms like product SKUs, error codes, or names that embeddings smear together), then rerank down to the 3–8 chunks that actually go in the prompt. A rerank step consistently outperforms retrieving directly at high precision, because a cross-encoder scores the query and passage jointly (full attention across both), whereas the bi-encoder embeddings used for the initial ANN search compress each text independently and lose fine-grained interaction signal.

Two practical reranking options. The first is Cohere's hosted Rerank 4, announced on 11 December 2025 and split into two models you choose between: rerank-v4.0-pro when you want the accuracy and rerank-v4.0-fast when you want the latency. Both are multilingual and take a 32K-token context, eight times the 4,096 tokens of the rerank-v3.5 generation they replace, which means a long chunk no longer has to be truncated before the cross-encoder sees it. The second is a local cross-encoder from sentence-transformers such as cross-encoder/ms-marco-MiniLM-L6-v2, if you need to keep reranking in-house:

from sentence_transformers import CrossEncoder

reranker = CrossEncoder("cross-encoder/ms-marco-MiniLM-L6-v2")

pairs = [(query, c["content"]) for c in candidates]
scores = reranker.predict(pairs)
ranked = sorted(zip(candidates, scores), key=lambda x: x[1], reverse=True)
top_chunks = [c for c, _ in ranked[:5]]

Hybrid search plus rerank is the combination that matters most in practice: vector search alone typically misses exact-match queries, and rerank alone is too slow to run over an entire corpus, so use ANN (or ANN + keyword, fused with reciprocal rank fusion) to get a fast, high-recall candidate set, then rerank for precision.

Generation with Citations

Assemble the prompt so the model can only answer from what you gave it, and so every claim can be traced back to a source chunk. Number the retrieved chunks and instruct the model to cite by number; store the chunk-ID-to-source mapping alongside the generation call so you can turn [2] into a real document link in the UI.

def build_prompt(query: str, chunks: list[dict]) -> str:
    context = "\n\n".join(
        f"[{i+1}] (source: {c['document_id']}, p.{c['page']})\n{c['content']}"
        for i, c in enumerate(chunks)
    )
    return f"""Answer the question using ONLY the numbered sources below.
Cite sources inline as [1], [2], etc. If the sources don't contain
the answer, say so explicitly instead of guessing.

Sources:
{context}

Question: {query}
Answer:"""

Send this to a current frontier chat model (or a comparable instruction-tuned open-weight model) with a low temperature (0–0.3) for factual QA — you want consistency, not creativity, here. Post-process the response to validate that every [n] citation actually exists in the numbered source list, and reject or regenerate answers that cite nothing when the question clearly required grounding. Keep the system prompt strict about refusing to answer outside the provided context — this single instruction is what prevents a RAG system from quietly reverting to the model's parametric (and unverifiable) knowledge when retrieval comes back thin.

Evaluation

Evaluate retrieval and generation separately — a low-quality answer with perfect retrieval is a prompting/generation bug, and a good answer built on the wrong sources is a retrieval bug, and conflating the two by only eyeballing final answers makes both nearly impossible to fix.

Build a labeled eval set of 50–200 realistic queries with the correct source chunk(s) identified by hand. Track standard IR metrics on the retrieval step: recall@k (did the right chunk make it into the top k?) and MRR (how high did it rank?). For end-to-end quality, RAGAS is the most widely used open-source framework and implements the metrics that matter most with an LLM-as-judge:

Run this eval suite on every change to chunking strategy, embedding model, or prompt template — these four numbers catch regressions that manual spot-checking reliably misses, especially context precision drops from over-broad top-k retrieval.

Common Pitfalls

Explore

More articles