RAG Chunking Strategies: 7 Methods Ranked for 2026

RAG chunking strategies shown as a document ribbon divided into ordered segments

Table of Contents

Here is a number worth sitting with: changing nothing but your chunking strategy can swing retrieval recall by around nine percentage points on the same corpus with the same model.

That is a larger effect than most model upgrades, and it costs nothing but attention. Yet chunking is consistently the stage teams configure once, badly, and never revisit – usually because it looks like plumbing rather than a decision.

Chunking splits documents into the units your retrieval system actually returns
A chunk is the unit of retrieval. Where you place the boundaries decides what your model ever gets to see.

Why Chunking Decides Your Retrieval Quality

A chunk is the unit of retrieval. The system does not return documents – it returns chunks, and whatever is in that chunk is what your model gets to reason with.

Which means chunking failures are invisible and fatal. Split a table down the middle and no query will ever retrieve a coherent version of it. Separate a definition from the term it defines and the definition becomes unfindable. Bundle four topics into one chunk and its embedding becomes a blurred average that matches nothing precisely.

None of this shows up as an error. It shows up as an assistant that is inexplicably bad at certain questions, and teams reliably misdiagnose it as a model problem. If retrieval quality is new to you, our guide to how retrieval-augmented generation works covers where chunking sits in the pipeline.

1. Fixed-Size Chunking

Split every N tokens or characters, usually with some overlap. The simplest chunking strategy and the default in most tutorials.

Use it for: uniform plain text, prototypes, and anything where you need a baseline fast.

Where it fails: everything structured. It cuts mid-sentence, mid-table and mid-thought with complete indifference. Overlap mitigates the damage without fixing the cause.

Worth using as your first pass precisely because it is a baseline – if a smarter strategy does not beat it on your evaluation set, you have learned something useful.

2. Recursive Character Splitting

Split on a priority list of separators – paragraphs first, then sentences, then words – falling back only when a chunk is still too large.

This is the sensible default and the strategy most production systems should start with. It respects natural boundaries where they exist without requiring you to describe your documents in advance.

Use it for: mixed corpora, general prose, anything where you want good-enough without configuration.

Where it fails: documents whose meaningful boundaries are not punctuation – contracts organised by clause number, spreadsheets, slide decks.

3. Structural Chunking

Split on the document’s own structure: markdown headings, HTML sections, contract clauses, slide boundaries, spreadsheet rows.

When your documents have real structure, this beats everything else in this list for a fraction of the effort. A chunk that corresponds to an actual section is coherent by construction, and you get natural metadata – the heading path – for free.

Use it for: documentation, technical manuals, legal documents, anything with headings.

Where it fails: unstructured prose, badly formatted PDFs, and documents where sections vary wildly in length. Very long sections still need a secondary split.

This is why parsing quality matters so much upstream – a pipeline that destroys layout cannot chunk structurally. Engines built around document understanding, like RAGFlow, exist largely to preserve the structure this strategy depends on.

4. Sentence and Sliding Window

Group a fixed number of sentences per chunk, with a window that overlaps neighbours so context bleeds across boundaries.

The overlap is the point. It means a fact stated at the end of one chunk is still partially present at the start of the next, which reduces the odds of a boundary cutting exactly through the answer.

Use it for: conversational transcripts, interviews, narrative text where meaning accumulates across sentences.

Where it fails: it inflates your index. Heavy overlap can multiply storage and add near-duplicate results that crowd out genuine variety.

5. Semantic Chunking

Embed each sentence, measure similarity between neighbours, and place boundaries where the topic shifts. Chunks end up varying in length because they follow meaning rather than counting.

Published comparisons put the gain at roughly nine percent recall over simpler chunking strategies – a real improvement, and one of the few places where the effect size is well documented.

The cost is genuine. You embed every sentence during ingestion, which makes indexing slower and more expensive. On a large corpus that is a real bill.

Use it for: unstructured prose where topics shift unpredictably and the corpus is small enough that ingestion cost is acceptable.

Where it fails: structured documents, where structural chunking gets you the same coherence for almost nothing.

6. Parent-Child (Small-to-Big)

Index small, precise chunks for matching, but return the larger parent section to the model. You search on specificity and answer with context.

This resolves the central tension in chunking. Small chunks retrieve accurately and lack context; large chunks carry context and match imprecisely. Parent-child gets both by decoupling the two jobs.

Use it for: almost any production system, and particularly technical documentation where a precise sentence needs its surrounding section to be useful.

Where it fails: it needs a store that handles the parent-child relationship, and it uses more context per answer. Our comparison of vector databases for RAG covers which handle this cleanly.

If you adopt one advanced strategy from this list, make it this one. The implementation cost is modest and the improvement is consistent.

7. Late Chunking and Contextual Retrieval

Two related recent techniques that attack the same problem: a chunk removed from its document loses the context that made it meaningful.

Late chunking inverts the usual order. Rather than splitting and then embedding, it embeds the whole document with a long-context embedding model first, then splits – so each chunk’s vector carries document-level context. A chunk saying “it increased by 12%” retains some trace of what “it” referred to.

Contextual retrieval takes a simpler route: before embedding, prepend a short generated description of where each chunk sits in its document. The chunk becomes self-describing. This is markedly better on queries involving pronouns and cross-references, which conventional chunking handles badly.

Use them for: corpora full of internal references, reports where meaning depends on earlier sections, anything where chunks read as fragments.

Where they fail: both cost more at ingestion. Late chunking requires a long-context embedding model, and contextual retrieval means a model call per chunk. On a large corpus that is a serious ingestion bill.

The Seven Compared

Strategy Best for Ingestion cost Retrieval quality
Fixed-size Baselines, uniform text Minimal Low
Recursive General default Minimal Decent
Structural Documents with headings Low High
Sentence / sliding Transcripts, narrative Low Decent
Semantic Unstructured prose High High
Parent-child Most production systems Low Very high
Late / contextual Reference-heavy corpora Very high Very high

A Worked Example on One Document

Take a forty-page employee handbook. Sections with headings, a few tables, some cross-references between policies.

Fixed-size at 512 tokens produces roughly eighty pieces, several of which begin mid-sentence and one of which contains the second half of the parental leave table and the first half of the expenses policy. Ask about leave entitlement and you get a confident answer built from a fragment.

Recursive splitting fixes the mid-sentence problem. Boundaries land on paragraph breaks. The table is still severed, because a table is not a paragraph.

Structural splitting on headings produces around thirty pieces, each corresponding to an actual policy. The table stays intact because it lives inside one section. Retrieval for a policy question now returns that policy, whole.

Adding parent-child means you index each paragraph separately for precise matching but return the full policy section. A query about a specific allowance matches the exact sentence and the model receives the whole policy around it.

Adding contextual retrieval handles the last failure. A paragraph reading “this does not apply to contractors” is meaningless in isolation; prepending its section context makes it retrievable and interpretable.

Four changes, each taking under an hour, and the difference between an assistant people stop trusting and one they use daily. Nothing about the model changed.

Matching Strategy to Document Type

Most real corpora contain several document types, and routing by type beats applying one approach to everything.

Document type Approach Why
Policies, handbooks Structural + parent-child Sections are already the right unit
Contracts Clause-level Clause numbering is the real structure
Technical docs Structural by heading Heading path doubles as metadata
Research papers Section-aware Abstract and references are distinct units
Meeting transcripts Sentence window with overlap Meaning accumulates across turns
Slide decks One slide per unit Slides are self-contained
Spreadsheets Row groups with headers repeated A row is meaningless without its headers
Support tickets One ticket per unit Natural boundaries already exist

The spreadsheet row is worth dwelling on, because it is the most commonly botched case. A row split away from its column headers is a list of values with no meaning attached. Repeating the header row inside every group costs a few tokens and rescues the entire document type.

How This Fits With Reranking

These two improvements are frequently framed as alternatives. They are not – they fix different failures and compound.

Good boundaries determine whether a coherent answer exists in your index at all. Reranking determines whether it surfaces above the near-misses. Neither compensates for the other: a reranker cannot promote a passage that was destroyed at ingestion, and clean boundaries do not help if the right passage sits eleventh in the results.

If you have limited time, add a reranker first – it is a configuration change rather than a re-index, so the feedback is immediate. Then fix your boundaries and re-index. Measuring after each step tells you which of your failures belonged to which stage, and that knowledge transfers to every future project.

How to Choose in Five Minutes

Work down this list and stop at the first match.

  1. Do your documents have headings or clear sections? Structural chunking, plus a size cap for long sections.
  2. Are they transcripts or conversations? Sentence windows with overlap.
  3. Are they unstructured prose, and is your corpus modest? Semantic chunking.
  4. Are chunks full of pronouns and back-references? Contextual retrieval on top of whatever you chose.
  5. Everything else, or unsure? Recursive splitting, then add parent-child.

Then layer parent-child over your choice regardless. It is compatible with every strategy above and improves most of them.

What Size Should Chunks Be?

The honest answer is that it depends on your questions, not your documents.

Narrow factual questions – a price, a date, a definition – want smaller chunks, roughly 200 to 400 tokens. Precision matters more than context.

Explanatory questions want larger chunks, roughly 500 to 1,000 tokens, because the answer spans several sentences.

Summarisation and comparison want parent-child, so you match narrowly and answer broadly.

The LangChain splitter documentation is a useful reference for implementations. Overlap of ten to twenty percent is a reasonable default where you are using it. More than that and you are mostly paying to store the same text repeatedly.

Do not tune this by intuition. Build a set of fifty real questions with known correct sources, measure how often the right chunk is retrieved, and change one variable at a time. Chunking is the one stage where measurement is easy and almost nobody does it.

Metadata Is Half the Job

Splitting text is only part of preparing a chunk. What you attach to it determines what you can do later, and it is nearly impossible to add retrospectively without a full re-index.

Five fields are worth storing on every piece without exception.

Source document and location. Filename, page, section. This is what makes citations possible, and citations are what make an assistant verifiable rather than merely fluent.

The heading path. If a piece came from a subsection, store the full trail of headings above it. This doubles as usable context and as a filter.

Document date. Essential for anything where currency matters. Without it you cannot prefer this year’s policy over the superseded one, and the system will cite both with equal confidence.

Document type. Contract, policy, transcript, spec. Enables routing at query time and makes it possible to filter a search to just the contracts.

Access scope. If different people should see different material, this must be attached at ingestion. Filtering by permission at query time is the only reliable approach, and it depends entirely on the metadata being there.

Teams that skip this step usually discover the gap six months in, when someone asks for a feature that would have been trivial and is now a re-index of the entire corpus.

Six Mistakes That Wreck Retrieval

1. Chunking before fixing parsing. No chunking strategy recovers from a PDF whose tables were flattened into character soup. Look at your extracted text first.

2. One strategy for every document type. A contract and a chat log do not want the same treatment. Route by type.

3. Dropping metadata. Store source, section, heading path and date with every chunk. It powers filtering and citations, and it is painful to backfill.

4. Enormous overlap. Fifty percent overlap doubles your index and fills results with near-duplicates.

5. Never re-chunking. Chunking is not permanent. If retrieval is poor, re-parse and re-index – it is a few hours, not a rewrite.

6. Tuning chunking instead of adding a reranker. Reranking often delivers more than any chunking change, and the two compound. Do both, but if you only do one, rerank.

Frequently Asked Questions

What is the best chunk size for RAG?

There is no universal answer. Roughly 200-400 tokens for factual lookup, 500-1,000 for explanatory content, and parent-child when you need both.

Does chunking matter more than the embedding model?

They set different ceilings. A better embedding model improves matching; better chunking improves what there is to match. Fix chunking first – it is cheaper and the failures are more severe.

Should I use overlap?

Ten to twenty percent for prose. None for structural chunking, where boundaries are already meaningful.

Is semantic chunking worth the cost?

On unstructured prose with a modest corpus, often yes. On structured documents, structural chunking gets you there for far less.

How do I know my chunking is bad?

Look at the retrieved chunks behind wrong answers. Truncated sentences, orphaned headings and split tables are all chunking failures, and they are obvious once you look.

Can I change chunking without re-indexing?

No. Changing how documents are split means re-parsing and re-embedding. Budget for it, and design your pipeline so it is a routine operation rather than a project.

Final Thoughts

Chunking is unglamorous, entirely unfashionable, and one of the highest-leverage decisions in a retrieval system. It sits early enough in the pipeline that every later stage inherits its mistakes, and it fails silently enough that nobody looks.

Start with recursive splitting to get a baseline. Move to structural chunking if your documents have structure, which most business documents do. Add parent-child. Measure after each change against a fixed question set.

That sequence takes an afternoon and will do more for your retrieval quality than a model upgrade costing considerably more.

Tags

Related Posts