Vector search and vector databases

Once you can embed text, the next problem is retrieval at scale: given a query vector, find the k closest vectors among millions. Exact search works until it doesn't. Approximate indexes trade a little recall for orders-of-magnitude latency and cost wins. Choosing the algorithm, the store and the filter strategy is the difference between a snappy RAG product and a $40k/month surprise.

This page is the map of the search layer. Chunking and generation live in RAG. Hybrid BM25 + dense fusion lives in advanced RAG. The embedding model chooses the coordinate system. The ANN index chooses how aggressively to approximate neighborhoods. Do not blame the store for a bad embedder, or retune HNSW when the chunks are garbage.

Concepts in this section

Read them in order. Each one builds on the one before.

Step	Concept	What it answers
1	Exact vs approximate search	When brute force is enough, and how to measure ANN recall
2	HNSW	How the default graph index works, and how to tune it
3	IVF, PQ and DiskANN	What to use when HNSW no longer fits in RAM
4	Filtered vector search	How tenancy and ACLs belong in the query, not the prompt
5	Choosing a vector store	pgvector, search engines or a dedicated store

A number to put on the whiteboard

10,000,000 vectors × 1,024 dims × 4 bytes ≈ 40 GB raw
HNSW edges often add another 1–2×  →  plan ~80–120 GiB RAM
Exact search: ~1e10 ops/query ≈ 200 ms on one generous core

You cannot brute-force 50M × 1536 at p99 < 50 ms. That is why ANN exists. Typical production goals: recall@10 ≥ 0.95, p95 under 20–50 ms, including filters. Measure on a held-out set with exact neighbors.

A sensible rollout: flat index plus evals while the corpus is small; HNSW in Postgres or a single node once latency demands it; dedicated cluster, PQ or disk when finance asks about a 256 GB box; hybrid lexical+dense when ID queries fail.

Walk the 10M × 1,024 math on a whiteboard, then say what changes if you truncate to 256 dims (~10 GB raw) and accept a measured recall drop. Mention pre- versus post-filtering unprompted. Interviewers listen for ACL mistakes.

Common mistakes

Skipping exact search, so you never know whether a miss is embeddings or approximation.
Post-filtering a tiny tenant on a shared graph and shipping empty or leaked results.
Enforcing ACLs only in the prompt.
Parameters tuned at 1M vectors still running at 20M with the same efSearch.
A model upgrade as an in-place UPDATE. It is a new index generation.

Key takeaways

Exact kNN is O(N · d). ANN buys latency with a recall trade-off you must measure.
HNSW is the default. Tune M, efConstruction, efSearch. Budget several× vector RAM.
IVF, PQ and DiskANN extend scale when memory or corpus size outgrows pure HNSW.
Filter at retrieval. Prefer pre-filter for selective ACLs. Never trust the prompt for tenancy.
Start with pgvector when Postgres is enough. Size memory from first principles.