Guide

Retrieval

Filtered vector search

Almost every real query is nearest neighbors among documents the caller may see. Filter at retrieval, never in the prompt.

56 of 155 · 2 min read

Almost every real query is "nearest neighbors among documents I am allowed to see" — filtered by tenant, ACL, language, product, updated_at and so on. Never rely on the LLM prompt to enforce access control. Filter at retrieval.

Concept

Before you start: ANN walks a graph or a set of clusters. Filters change which nodes that walk is allowed to land on. Get this wrong and tenant B's chunks leak into tenant A's answer.

Pre-filter vs post-filter

Pre-filtering applies the metadata predicate during graph or cluster traversal so candidates already satisfy filters. Post-filtering retrieves top-k then drops non-matching rows (and may over-fetch to compensate).

Trade-off

Pre-filter when selectivity is high (one tenant among thousands) so you do not waste the graph walk on forbidden vectors. Post-filter can work for light filters if you over-fetch (for example ask for 50, keep 10 after filter). Highly selective post-filter on a tiny slice of the corpus can return empty or low-quality sets — the ANN never "knew" to search inside the slice.

Multi-tenancy patterns

  1. Shared index + tenant_id filter — simple ops, careful about noisy neighbors and filter performance.
  2. Per-tenant indexes or collections — strong isolation, higher ops cost, great when tenants differ wildly in size.
  3. Hybrid — large tenants get dedicated indexes; small tenants share a pool.

Example

Tenant A has 50k chunks; tenant B has 8M. A shared HNSW graph spends most of its structure on B. Queries for A with a post-filter may walk neighbors that belong to B and then discard them, hurting both latency and recall for A. Either pre-filter effectively, shard by tenant for large accounts, or maintain a separate collection per large tenant.

Payload indexes (B-trees or inverted indexes on metadata) matter as much as the vector index when filters are selective. A beautiful HNSW with a slow tenant_id scan still misses SLOs.

Over-fetch heuristics that worked for 10% filter selectivity fail at 0.1%. Watch filter empty-result rate as its own metric. A silent empty set becomes a hallucinated answer if the generator has no "I don't know" path.

Walking through a leaky filter

Query: tenant=A, k=10, post-filter
ANN returns 10 neighbors: 9 from B, 1 from A
After filter: 1 hit
Generator sees one thin chunk and invents the rest

Same query, pre-filter or a dedicated A index:

ANN walks only A's 50k vectors
Returns 10 on-tenant neighbors
Generator has a real top-k

The embedder did not change. The filter strategy did.

Common mistakes

  • Enforcing ACLs only in the prompt. The model can ignore them.
  • Post-filtering a 0.1% slice and treating an empty result as "no one asked."
  • One shared graph for a 50k tenant and an 8M tenant with no pre-filter.

Check yourself

  1. Why is a prompt that says "only use tenant A's docs" not an ACL? The model can ignore it. The retrieval query must exclude other tenants.
  2. When do you prefer a dedicated index per tenant? When tenants differ wildly in size, or isolation matters more than ops simplicity.

Key takeaways

  • Filter at retrieval. Never trust the prompt for tenancy.
  • Pre-filter selective predicates. Post-filter only light ones, and over-fetch.
  • Large tenants often need their own collection.
  • Watch empty-result rate. An empty retrieval is how grounded systems start inventing.

Further learning

RAG is where filtered hits become an answer. Next, choosing a vector store.

Vector search and vector databases

Approximate nearest neighbor algorithms (HNSW, IVF, PQ) and choosing between pgvector and dedicated stores.

HNSW

A layered graph. Search walks sparse highways first, then a dense bottom layer. Tune M, efConstruction and efSearch.

Retrieval-augmented generation

The core RAG pipeline: ingest, chunk, embed, retrieve, generate with citations.