Chunking for RAG

Chunking splits each document into the passages you retrieve. Chunk boundaries decide what can be found. Each chunk gets one embedding, which is a single point in space. If two unrelated facts share a chunk, the point sits between them and matches neither well. If a fact is split across two chunks with no overlap, neither chunk holds the whole fact.

The big idea

Cut on structure first, and on size second. Headings, paragraphs, HTML sections and code symbols are natural boundaries. A fixed window cut every 500 characters is convenient, but it can slice a sentence or a word in half.

Look at this naive cut. It makes chunks of 30 characters with an 8-character overlap:

def chunk_text(text, size, overlap):
    """Cut text into pieces of `size` characters. Each piece repeats the last `overlap` characters of the one before."""
    chunks = []
    start = 0
    while start < len(text):
        chunks.append(text[start:start + size])
        start += size - overlap
    return chunks


doc = "Refunds are allowed within 30 days of delivery. Items must be unused."
for n, chunk in enumerate(chunk_text(doc, size=30, overlap=8)):
    print(n, repr(chunk))

Output:

0 'Refunds are allowed within 30 '
1 'thin 30 days of delivery. Item'
2 'ry. Items must be unused.'
3 'ed.'

Chunk 1 starts in the middle of the word "within." Chunk 2 starts in the middle of "delivery." The fact "items must be unused" is intact only because the overlap happens to cover it. Structure-aware splitting avoids this, because it cuts at sentence or heading boundaries.

Walking through the numbers

The step between chunks is size - overlap. With 30 and 8, each new chunk starts 22 characters after the previous one. The overlap is 8 out of 30 characters, which is about 27%. A common text setting is 10 to 20% overlap, so this demo is wider than you would ship.

For a real document, the same arithmetic applies in tokens. A 2,400-token policy with clear headings might become about six chunks of roughly 400 tokens, with about 50 tokens of overlap. Each chunk stores its document ID, section and URL.

Three decisions

• Size. A few hundred tokens, often about 200 to 500, is a common start. Smaller chunks are precise but starve the generator of nearby context. Larger chunks carry more meaning but blur the embedding. Keep the size under the embedder's maximum, or it will truncate silently.
• Overlap. About 10 to 20% helps facts on a boundary survive in at least one chunk.
• Context prefix. Put the document and section in front of each chunk, such as Document: Refund policy | Section: Eligibility. A chunk retrieved alone then still says where it came from.

Common mistakes

• Fixed character windows. They cut words and sentences. Split on structure first.
• One size for everything. A code file and an HR policy need different rules.
• No overlap at all. Facts on a boundary can vanish from every chunk.
• Broken parsing that looks like chunking. Multi-column PDFs, flattened tables and cookie banners repeated in every chunk all hurt retrieval. Fix the parser before you tune the size.

Check yourself

• Why does overlap help? A fact that sits on a boundary appears whole in at least one of the two chunks.
• Why might a 30-character chunk fail on a real policy? It is too small to hold the rule and its exception together, so the generator sees half the policy.

Key takeaways

• Chunk boundaries decide what retrieval can find.
• Split on headings and paragraphs before you split on length.
• Use about 200 to 500 tokens and 10 to 20% overlap as a starting point, then tune with evals.
• Prepend the document and section so each chunk is self-describing.

Further learning

Next, see how the retrieved chunks become a grounded, cited answer in grounded answers.