RAG chunking strategies compared: fixed, recursive, structural, semantic and parent-child

RRetrieval-augmented generation (RAG) gives a language model access to your documents: you split them into pieces, index the pieces, retrieve the most relevant ones for each question and put them in the prompt. Teams often spend weeks choosing an embedding model and a vector database, then split documents every 500 characters and wonder why answers are poor.

Chunking, the way you split documents, is one of the highest-impact decisions in a RAG system. A chunk is the unit your retriever can find. If the answer is split across two chunks, or buried in a chunk dominated by unrelated text, no embedding model can rescue it. This guide compares five chunking strategies, explains when each works, and gives a method for choosing on your own data.

What a good chunk looks like

A good chunk is:

• Self-contained: it makes sense without the text around it.
• Focused: it is about one topic, so its embedding represents that topic clearly.
• Big enough to contain an answer, including the context needed to interpret it.
• Small enough to be precise, so retrieved text is mostly relevant and fits in the prompt alongside others.

These goals pull in opposite directions. Small chunks are precise but lose context; large chunks keep context but dilute relevance. Every strategy below is a different way of managing that trade-off.

Strategy 1: fixed-size chunks

Split text every *N* tokens, usually with an overlap of 10 to 20% so sentences cut at a boundary appear in both neighbouring chunks.

• Pros: trivial to implement, predictable sizes, works on any text.
• Cons: ignores meaning. Cuts mid-sentence, mid-table and mid-list; mixes the end of one section with the start of another.
• Use when: you need a baseline, or the text has no structure (transcripts, plain prose).

Strategy 2: recursive splitting

Split on the largest natural separator first (paragraph breaks), and only if a piece is still too large, split on smaller ones (lines, then sentences, then words). This is the default in several popular RAG libraries.

• Pros: respects paragraphs and sentences, still simple, good general default.
• Cons: unaware of document structure such as headings and sections.
• Use when: general prose documents without reliable markup.

Strategy 3: structure-aware chunking

Use the document's own structure: headings in Markdown or HTML, sections in PDFs, functions and classes in code, rows or records in tables. Each chunk is a section, sub-section or logical unit, split further only if it is too long. Prepend the heading path ("Billing > Refunds > Annual plans") to each chunk.

• Pros: chunks match how authors organised information; heading paths add valuable context.
• Cons: requires parsing; messy PDFs and inconsistent formatting make it harder.
• Use when: documentation, policies, manuals, knowledge bases, codebases. Usually the biggest single improvement over fixed-size chunking.

Strategy 4: semantic chunking

Embed each sentence, then start a new chunk wherever the similarity between consecutive sentences drops sharply, signalling a topic change.

• Pros: adapts to content without needing markup.
• Cons: more expensive to compute, sensitive to thresholds, and gains over good structural or recursive chunking are often modest and inconsistent.
• Use when: long unstructured text with clear topic shifts, such as meeting transcripts or interviews.

Strategy 5: parent-child (small-to-big) retrieval

Index small chunks for precise matching, but when one matches, return its larger parent (the full section or a window of surrounding text) to the model.

• Pros: combines precise retrieval with enough context to answer.
• Cons: more storage and complexity; parents can be large and consume prompt space.
• Use when: answers need surrounding context, such as definitions, exceptions or multi-paragraph explanations.

The strategies side by side

Add context to every chunk

Whatever strategy you choose, chunks lose context when they are cut out of a document. A short sentence like "This limit increases to 50 for annual plans" is useless on its own: which limit, which product?

Two fixes help a lot:

• Prepend metadata: document title, section heading path, date and product to every chunk before embedding.
• Contextual chunk descriptions: use a model to write one or two sentences situating each chunk within its document, and prepend them before indexing. Anthropic described this approach as "contextual retrieval" in 2024 and reported large reductions in retrieval failures, especially when combined with keyword search and reranking.

Retrieval settings matter as much as chunking

• Hybrid search. Combine embedding similarity with keyword search (such as BM25). Keywords catch exact terms such as product codes, error messages and names that embeddings handle poorly.
• Reranking. Retrieve generously (say, 20 to 50 candidates), then use a reranker model to choose the best handful.
• Number of chunks. More chunks raise recall but add noise and cost. Order them by relevance, keeping the best near the question; see lost in the middle.

Special content: tables, code and PDFs

Some content breaks every general-purpose strategy and needs its own handling:

• Tables. Splitting a table by tokens separates values from their column headers, making them meaningless. Keep small tables whole, and for large ones create one chunk per row or group of rows with the header repeated in each.
• Code. Split by function, class or module using a parser, and include the file path and signature in each chunk. Line-based splitting cuts functions in half.
• PDFs. Text extraction often scrambles columns, headers, footers and tables. Inspect extracted text before tuning chunking; fixing extraction often helps more than any chunking change.
• Slides and short-form content. One slide or card per chunk usually works better than merging several into a fixed size.

How to choose: test on your own questions

Chunking advice is a starting point. The right choice depends on your documents and questions, so measure.

1. Build a retrieval eval set. Write 50 to 100 real questions and, for each, mark the passage or passages that contain the answer. See build an eval set in an afternoon.
2. Measure retrieval separately from generation. For each strategy, check recall at k: does the right passage appear in the top 5 or top 10 results? Generation quality cannot exceed retrieval quality.
3. Then measure answers end to end, including whether citations point to the right passages; see prompting for citations that check out.
4. Try two or three chunk sizes for your chosen strategy, for example 200, 400 and 800 tokens.
5. Read the misses. Most retrieval failures have visible causes: answers split across chunks, missing context, tables mangled by PDF extraction.

A sensible starting point for many document collections: structure-aware chunks of a few hundred tokens with heading paths prepended, hybrid search, a reranker and parent expansion for the top results. Then let your eval set tell you what to change.

Frequently asked questions

What is chunking in RAG?

Chunking is splitting documents into smaller pieces that are indexed and retrieved individually. The retriever can only return whole chunks, so the way documents are split strongly affects whether the right information reaches the model.

What is the best chunk size for RAG?

There is no universal best size. A few hundred tokens is a common starting point. Smaller chunks improve precision but lose context; larger chunks keep context but dilute relevance. Test two or three sizes on your own questions.

Should chunks overlap?

For fixed-size and recursive chunking, an overlap of around 10 to 20% reduces the chance of cutting an answer in half. Structure-aware chunking needs less overlap because it splits at natural boundaries.

Is semantic chunking better than fixed-size chunking?

Sometimes, but not reliably. Semantic chunking costs more and its gains over good recursive or structure-aware chunking are often small. Measure retrieval recall on your own data before adopting it.