The RAG ingestion pipeline

Ingestion is the offline half of RAG. It reads your documents, turns them into searchable pieces and keeps those pieces in step with the source. Nothing in the question path can be better than what ingestion stored. If the index holds last month's refund policy, the best prompt in the world will quote it confidently.

The big idea

Ingestion is an assembly line with seven steps:

1. Parse. Extract clean text from PDFs, HTML, Markdown, tickets or wiki exports. Keep headings and tables where you can.
2. Clean. Strip navigation, boilerplate and repeated footers.
3. Chunk. Split the text into passages. The next concept page covers this.
4. Attach metadata. Store the source ID, title, URL, section, updated_at, tenant and access tags.
5. Embed. Use the same embedding model you use at query time, and record its version.
6. Index. Write the vectors and metadata to pgvector or another vector store.
7. Sync. On create, update and delete, re-chunk the affected document, re-embed and upsert or delete.

Step 7 is the one teams skip. Stale indexes are one of the top three production failures in RAG. Keep a content hash per document, so you can spot drift: "the wiki says X, but the index still has last month's X."

A library catalog is a fair picture. The catalog must add new books, update changed ones and remove withdrawn ones. The analogy breaks here: a catalog does not have to answer in seconds for a thousand users at once.

Walking through it

Take a refund policy with six chunks. An editor changes section 2 only. With a surgical sync:

policy refund-v3 changed: section 2 (eligibility)
  re-chunk:  refund-v3 -> 6 chunks, 1 text changed
  re-embed:  1 chunk
  upsert:    1 chunk, same chunk ID
  untouched: 5 chunks

Without a content hash, the sync may re-embed all six. That costs more and makes the update window longer. The reverse matters too: if the old section 2 chunk is never deleted, search can return both the old and new eligibility rules.

Access control belongs in ingestion, too. Store the ACL tags on every chunk, and enforce them in the retrieval query. A system prompt that says "only use documents this user may see" is not a security boundary.

Common mistakes

• Insert-only sync. Deleted or renamed documents keep answering questions.
• Mixing embedding model versions. Vectors from two models do not compare cleanly. Re-embed everything when you change the model.
• Filtering access after retrieval. Post-filtering can leak the existence of documents, and it can hide good results. Filter in the query.
• Trusting the parser. Multi-column PDFs and flattened tables produce fluent but wrong text.

Check yourself

• Why keep a content hash per document? It lets you re-embed only the chunks that changed, and it reveals when the index no longer matches the source.
• Why must query-time embeddings use the same model as ingestion? Vectors from different models live in different spaces, so distances between them mean nothing.

Key takeaways

• Ingestion is parse, clean, chunk, tag, embed, index and sync.
• The sync must handle updates and deletes, not just inserts.
• Store ACL tags on each chunk and enforce them in retrieval.
• Record model and parser versions so you can re-run the pipeline when they change.

Further learning

Next, see how the chunking step decides what can be found, in chunking for RAG.