Document Chunking for RAG: How to Split Text to Get Better Answers from LLMs

Chunking is the splitting of documents into smaller blocks of text before indexing them in a RAG system, short for Retrieval-Augmented Generation. This choice determines what the language model receives as context: badly cut chunks produce incomplete or wrong answers even with an excellent LLM.
Why Chunking Matters
In a RAG system the user's question is compared with the indexed chunks and only the most relevant reach the model. A chunk that is too large mixes different topics and its embedding becomes generic; one that is too small loses the context needed to understand what a sentence refers to. In both cases retrieval gets worse, and the answer with it.
The Main Strategies
- Fixed size: chunks of constant length in tokens or characters. Simple and fast, but it can break sentences and tables.
- Recursive: tries to split by paragraphs, then by sentences, and only at the end by length. A good general starting point.
- Structure-based: follows the document's headings, sections and clauses. Suited to manuals, contracts and technical documentation.
- Semantic: finds topic changes by comparing the embeddings of neighbouring sentences. More expensive, useful for loosely structured text.
- Hierarchical: indexes small chunks for search and returns to the model the larger section that contains them.
Size and Overlap
There is no size that fits everything. As a starting point many projects use chunks of 200 to 800 tokens with a 10-20 percent overlap, so a sentence straddling two chunks is not lost. Specific questions, such as a code or a deadline, work better with short chunks; questions that need reasoning across several passages need larger ones. The choice has to be checked on your own documents.
Metadata and Chunk Context
An isolated chunk often loses essential information: which product it is about, which version or date it refers to. Store source metadata with each chunk, such as document title, section, date and confidentiality level, and use it to filter the search. One effective technique is to prepend to each chunk a short sentence that places it within the document before computing its embedding.
Tables, PDFs and Scanned Documents
Many problems start before chunking, in text extraction. Multi-column PDFs, repeated headers and tables need cleaning and conversion to a structured format such as Markdown. Tables should not be broken: keep them whole or repeat the header row in each part. Scans need a text recognition step with a quality check.
How to Evaluate the Choice
- Prepare a set of real questions with the document passage that contains the answer.
- Measure how often the correct passage appears among the first retrieved results.
- Assess how faithful the generated answers are to the sources.
- Compare two or three configurations and change one parameter at a time.
- Repeat the check when the documents or the embedding model change.
Common Mistakes
The most common mistakes are using the same configuration for different document types, ignoring the structure already present in the text, indexing duplicate or outdated versions and not planning index updates when a document changes. Choosing the size based only on the model's context window is also misleading: more retrieved text does not mean better answers.
Key takeaway: in RAG, answer quality depends on retrieval, and retrieval depends on how the documents are cut. Starting from a split that respects structure, adding metadata and measuring results on real questions is the most reliable way to improve.