Implementation Guides

Vector Search Implementation Guide

A hands-on guide to shipping vector similarity search: choosing an embedding model, HNSW vs IVF vs PQ, pgvector 0.8.6 iterative scans, hybrid BM25 fusion with RRF, Cohere Rerank 4, and recall tuning.

Gabriele Masetti · 2026-08-18

Scope of this guide

What follows is a build guide, not a theory primer. It assumes you already know what an embedding is and why cosine similarity works, and walks through the decisions you actually have to make to ship vector search in production: which embedding model, how to store and index vectors, which index type, which database, how to filter by metadata, how to layer in keyword search, and how to tune recall without guessing.

Choosing an embedding model

Three variables matter more than benchmark leaderboard position: dimensionality, cost, and whether the model matches your domain (general text vs. code vs. multilingual).

• OpenAI text-embedding-3-small — 1,536 dimensions, priced around $0.02 per 1M input tokens. Good default for general-purpose semantic search where cost and latency matter more than marginal accuracy.

• OpenAI text-embedding-3-large — 3,072 dimensions, roughly 6.5x the price of -small. Both models expose a dimensions parameter so you can truncate the output (e.g., request 1,024 instead of 3,072) via Matryoshka representation learning, trading a little accuracy for smaller storage and faster search without retraining anything.
• Cohere embed-v4 / open models like BAAI's BGE and Nomic Embed — worth benchmarking if you need multilingual coverage or want to self-host to avoid per-call API costs at high volume.

Practical rule: pick the smallest embedding dimension that hits your recall target on a held-out set of real queries. Every dimension you carry multiplies storage and index memory across every vector in the corpus, so a 3,072-dim model on 50M documents is a materially different infrastructure bill than 1,536.

Lock the model choice before you write a single row — embeddings from different models (or different dimensions truncations of the same model) are not comparable, and you cannot mix them in one index.

Generating and storing vectors

Chunking happens before embedding and determines retrieval quality as much as the model does. For prose, 200–500 token chunks with 10–20% overlap is a reasonable starting point; for code or structured docs, chunk on natural boundaries (functions, sections) rather than fixed token counts.

Batch-generate embeddings rather than calling the API per document:

import json
import openai
import psycopg

client = openai.OpenAI()

def embed_batch(texts: list[str], model: str = "text-embedding-3-small") -> list[list[float]]:
    resp = client.embeddings.create(model=model, input=texts, dimensions=1536)
    return [d.embedding for d in resp.data]

BATCH_SIZE = 96  # keep batches well under the API's request size/token ceiling

with psycopg.connect(DATABASE_URL) as conn:
    with conn.cursor() as cur:
        for i in range(0, len(chunks), BATCH_SIZE):
            batch = chunks[i : i + BATCH_SIZE]
            vectors = embed_batch([c.text for c in batch])
            cur.executemany(
                "INSERT INTO documents (content, metadata, embedding) VALUES (%s, %s, %s)",
                [(c.text, json.dumps(c.meta), v) for c, v in zip(batch, vectors)],
            )
    conn.commit()

Store the raw text alongside the vector, not just the vector — you need it for reranking, display, and re-embedding when you upgrade models. Store a model and embedding_version column too; you will change embedding models eventually, and mixed-model rows silently corrupt similarity scores if you don't track provenance.

Index types: HNSW, IVF, and PQ

Three algorithms cover almost every production system:

• HNSW (Hierarchical Navigable Small World) builds a multi-layer proximity graph. Query complexity is roughly logarithmic in corpus size, and it gives the best speed/recall tradeoff for most workloads without much tuning. Cost: higher build time and more memory than IVF.
• IVF (Inverted File Index) clusters vectors with k-means into lists cells and searches only the probes nearest cells at query time. Faster to build, lower memory, but recall degrades linearly as you cut probes, and it needs a representative training sample before you can build it.
• PQ (Product Quantization) compresses vectors by splitting each into subvectors and quantizing each subvector independently — it's a compression technique layered onto IVF (as IndexIVFPQ in FAISS) rather than a standalone index, trading accuracy for large memory savings at billion-vector scale.

Rule of thumb: start with HNSW. Only reach for IVF/PQ when the corpus is large enough (tens of millions+ of vectors) that HNSW's memory footprint becomes the bottleneck, or when you're using FAISS directly and need composite indexes like IVF4096,PQ64.

pgvector vs. FAISS vs. Pinecone/Weaviate/Qdrant

These aren't interchangeable — they solve different problems.

• pgvector is a Postgres extension: your vectors live in the same database as your relational data, so joins, transactions, and metadata filters are ordinary SQL. Best choice when you already run Postgres and don't need to scale past what a well-tuned Postgres instance can handle (which, with HNSW and read replicas, is a lot further than people assume).
• FAISS is a library, not a database — no persistence, replication, filtering, or network API out of the box. You embed it inside your own service and handle durability and filtering yourself. Use it when you want maximum control over index composition (e.g., custom IVFPQ configurations) or are doing offline/batch similarity search rather than serving live queries.
• Pinecone, Weaviate, Qdrant are managed or self-hostable vector databases with built-in filtering, hybrid search, multi-tenancy (namespaces/collections), and horizontal scaling. Reach for these when vector search is a first-class product feature at scale, and you don't want to operate the index tier yourself.

Decision shortcut:

pgvector in practice

CREATE EXTENSION IF NOT EXISTS vector;

CREATE TABLE documents (
    id BIGSERIAL PRIMARY KEY,
    content TEXT NOT NULL,
    metadata JSONB DEFAULT '{}',
    embedding_model TEXT NOT NULL,
    embedding vector(1536) NOT NULL
);

CREATE INDEX documents_embedding_hnsw
    ON documents
    USING hnsw (embedding vector_cosine_ops)
    WITH (m = 16, ef_construction = 64);

CREATE INDEX documents_metadata_gin
    ON documents USING gin (metadata jsonb_path_ops);

m (default 16, range 2–100) controls graph connectivity; ef_construction (default 64, must be ≥ 2×m) controls build-time search thoroughness. Both are build-time only — changing them means rebuilding the index.

Metadata filtering

Every real deployment needs to combine "semantically similar to X" with "and belongs to tenant Y" or "and created after date Z." Two approaches:

1. Pre-filter with a regular index, then vector search — works well when the filter is highly selective (e.g., tenant isolation) and your database can intersect a B-tree/GIN scan with the vector index efficiently.
2. Post-filter after the ANN search — retrieve more candidates than you need (e.g., top 50 instead of top 10) and filter afterward. That is the default behaviour of a naive HNSW query, and it can silently under-return results when a filter is selective, because the graph traversal may exhaust its candidate list before finding enough matches.

pgvector 0.8.0 (October 2024) added iterative index scan, which lets a filtered HNSW/IVFFlat query keep expanding the graph traversal in batches instead of giving up early, meaningfully improving recall on selective WHERE clauses. The knobs are hnsw.iterative_scan and ivfflat.iterative_scan, and they are unchanged in the current 0.8.x series — 0.8.6, released 29 July 2026. In managed databases, Pinecone indexes metadata automatically on upsert with no separate schema step, and recommends namespaces over metadata filters for tenant-scale isolation because a namespace-scoped query is cheaper than scanning-and-filtering a shared index.

Hybrid search: BM25 + dense vectors

Dense embeddings miss exact matches — product SKUs, error codes, proper nouns the embedding model undertrained on. Keyword search (BM25, or Postgres full-text search) catches those but misses paraphrase and synonymy. Combine both and fuse with Reciprocal Rank Fusion (RRF):

score(d) = Σ 1 / (k + rank(d))

summed across each retrieval method's rank for document d, with k = 60 as the conventional default. RRF operates on ranks rather than raw scores, which sidesteps the problem that BM25 scores and cosine similarities live on incompatible scales — it needs no tuning and is the default fusion method in Elasticsearch and Qdrant.

A hand-rolled RRF in SQL, combining pgvector similarity with Postgres full-text search:

WITH vector_hits AS (
    SELECT id, row_number() OVER (ORDER BY embedding <=> $1) AS rnk
    FROM documents
    ORDER BY embedding <=> $1
    LIMIT 50
),
text_hits AS (
    SELECT id, row_number() OVER (ORDER BY ts_rank_cd(content_tsv, q) DESC) AS rnk
    FROM documents, plainto_tsquery('english', $2) q
    WHERE content_tsv @@ q
    LIMIT 50
)
SELECT COALESCE(v.id, t.id) AS id,
       COALESCE(1.0 / (60 + v.rnk), 0) + COALESCE(1.0 / (60 + t.rnk), 0) AS rrf_score
FROM vector_hits v
FULL OUTER JOIN text_hits t ON v.id = t.id
ORDER BY rrf_score DESC
LIMIT 10;

Weaviate implements the same idea natively via an alpha parameter (0 = pure BM25, 1 = pure vector, default 0.5) with two selectable fusion algorithms, rankedFusion and the newer relativeScoreFusion. Qdrant exposes hybrid queries through its Query API combining dense and sparse (BM25-style) vectors natively with RRF. If your database supports hybrid search natively, use that instead of hand-rolling it — it's less code to maintain and usually faster.

Add a reranking stage for anything user-facing: take the top 20–50 hybrid candidates and rerank with a cross-encoder. Cohere's Rerank 4, released in December 2025, is the current hosted option, in two sizes — rerank-v4.0-pro for quality and rerank-v4.0-fast for latency and throughput — both with a 32,768-token context window against the 4,096 of the older rerank-v3.5, and both multilingual across 100+ languages. The wider window matters when the candidates are long: contracts, manuals, transcripts. A self-hosted model such as bge-reranker-v2-m3 is the alternative when the documents cannot leave your network. Budget roughly 100–300ms of added latency per query either way, and measure it on your own document lengths rather than trusting the range. Reranking consistently improves NDCG over first-stage retrieval alone because cross-encoders score the query and document jointly instead of comparing precomputed vectors.

Recall tuning

Recall is a query-time knob, separate from the index you built:

• HNSW: raise ef_search (default varies by system) to trade query latency for recall; it can be set per-session or per-transaction in Postgres via SET hnsw.ef_search = 100;.
• IVFFlat: raise probes — start around lists / 10 for corpora up to ~1M rows, and closer to sqrt(lists) for larger ones, adjusting via SET ivfflat.probes = 10;.

Measure recall against a ground-truth set, not vibes: run a brute-force exact search (flat index) on a sample of queries, compare to your ANN index's top-k, and compute recall@k directly. Re-check after any reindex, model change, or quantization change — recall regressions are silent until a user notices bad results.

If storage is the bottleneck rather than latency, pgvector's halfvec type (16-bit floats instead of 32-bit) cuts storage in half with negligible recall impact, and binary quantization via the bit type cuts storage roughly 32x at a real recall cost that a rerank stage can partially recover.

Batch vs. realtime ingestion

Two ingestion patterns, often needed together:

• Batch: nightly or hourly jobs that (re)embed a full corpus or a changed subset. Use this for the initial backfill and for periodic reprocessing (new embedding model, chunking strategy change). Batch APIs (e.g., OpenAI's batch endpoint) run at roughly half the per-token cost of synchronous calls in exchange for turnaround measured in hours, which is the right tradeoff for anything that isn't user-facing in real time.
• Realtime: a document is created or edited and needs to be searchable within seconds — support tickets, chat messages, freshly published articles. Embed and upsert synchronously (or via a fast queue) on the write path, and make sure your index type tolerates frequent small inserts — HNSW handles incremental inserts natively; IVF indexes trained on a stale sample degrade until retrained.

For change-heavy sources (a database feeding your vector store), a CDC pipeline (e.g., Debezium reading Postgres WAL) into a queue that triggers re-embedding is more reliable than polling for "updated_at > last_run," especially once you need to also handle deletes.

Production checklist

• Track embedding_model and a version per row; never mix models silently in one index.
• Set an explicit recall target (e.g., recall@10 ≥ 0.9 against exact search) and test it after every index or model change.
• Over-fetch before filtering or reranking (ask for 3–5x the final k) rather than assuming the ANN index will respect a selective WHERE clause at exactly k.
• Add hybrid (BM25 + dense) search before you assume you need a bigger embedding model — most "bad recall" complaints are exact-match misses, not semantic ones.
• Monitor index build time and query p99 latency separately; they degrade differently as the corpus grows and require different fixes (index type change vs. ef_search/probes tuning).