Chunking and indexing

Chunking is splitting documents into retrievable pieces. It is unglamorous and it matters more than which embedding model you picked.

Why chunk at all

Two reasons. Whole documents are too big to put several of into a context window. And a single vector for a long document averages away everything specific in it, so it matches everything vaguely and nothing well.

The size trade

Too small, say a sentence, and the chunk lacks the context to be understood. "It must be renewed annually" is useless without knowing what it is.

Too large, say a whole chapter, and the vector is a blur. It also wastes context window on mostly irrelevant text.

A common working range is 200 to 500 words, or roughly 300 to 800 tokens. Treat that as a starting point to measure from, not a rule.

Overlap prevents split answers

Include the last sentence or two of the previous chunk at the start of the next. Ten to twenty percent overlap is typical.

Without it, an answer that straddles a boundary is in neither chunk completely, and retrieval returns half of it.

Split on structure, not character count

This is the single biggest improvement available.

Splitting every 500 characters cuts through the middle of sentences and tables. Splitting on document structure keeps meaning intact:

• Markdown: split on headings, keep each section together
• Code: split on function or class boundaries
• HTML: split on semantic sections
• Transcripts: split on speaker turns or topic changes

If your source has headings, use them. They exist because a human already decided where the ideas divide.

Carry the heading into the chunk

Prepend the document title and section heading to each chunk before embedding:

Billing > Refunds > Timelines

Refunds are processed within five working days...

Now a chunk that says "it takes five working days" is findable by someone asking about refund timing. This is cheap and disproportionately effective.

Store metadata alongside

Keep source document, section, URL, and last updated date with every chunk.

You need the source to cite it, the date to filter stale content, and the section to give the model orientation. Adding this later means re-indexing everything.

Re-indexing is normal

Content changes. Plan for updates from the start: track which chunks came from which document, so changing a document replaces exactly its chunks rather than duplicating them.

Duplicate chunks from a partial re-index are a common and confusing failure. The system returns three near identical passages and the answer looks strangely repetitive.

How to tell if it is working

Take twenty real questions with known answers. For each, check whether the correct passage appears in the top few results.

If it does not, no amount of prompt engineering downstream will save the answer. Fix retrieval first. That measurement is the subject of when RAG fails.