RAG Chunking Strategies: From Fixed-Size to Late Chunking

Most RAG tutorials spend 80% of their time on embedding models and vector databases. Chunking gets a footnote. That's backwards.

Your retrieval pipeline can only return what it indexed. If a chunk straddles a topic boundary, the embedding is muddied. If chunks are too large, the relevant sentence drowns in noise. If they're too small, you lose the context that makes the sentence meaningful. The embedding model is not magic — garbage in, garbage out.

Five strategies cover most production use-cases:

• Fixed-size — simplest baseline, character or token count
• Recursive — structure-aware splitting on a priority list of separators
• Semantic — split on embedding similarity shifts
• Hierarchical — dual-granularity index (summary + detail)
• Late chunking — embed the whole document first, then pool

Each makes a different bet about where meaning lives in your corpus.

Fixed-size chunking slices text every N tokens (or characters) with an optional overlap. It is fast, deterministic, and model-agnostic.

from langchain.text_splitter import TokenTextSplitter

splitter = TokenTextSplitter(
    chunk_size=512,
    chunk_overlap=64,
    encoding_name="cl100k_base",  # tiktoken encoder for GPT-4 / text-embedding-3
)
chunks = splitter.split_text(document)

Why overlap exists: A sentence that starts at token 510 would otherwise be split between two chunks, severing the context. Overlap of 10–15% of chunk size is the common heuristic.

Gotcha: Character-based splitting and token-based splitting diverge badly on non-Latin scripts and code. Always use the same tokenizer as your embedding model — a 512-token budget for text-embedding-3-small is exactly 512 cl100k tokens, not 512 characters.

Fixed-size works surprisingly well on homogeneous corpora (e.g., news articles, product descriptions). It breaks down on structured documents where a logical section is 50 tokens or 2 000 tokens depending on where you happen to cut.

Recursive chunking tries a priority-ordered list of separators and only falls back to the next one when the resulting chunk would still be too large. LangChain's RecursiveCharacterTextSplitter defaults to ["\n\n", "\n", " ", ""] — paragraphs first, then lines, then words, then characters.

from langchain.text_splitter import RecursiveCharacterTextSplitter

splitter = RecursiveCharacterTextSplitter(
    chunk_size=1000,
    chunk_overlap=100,
    separators=["\n\n", "\n", ".", " ", ""],
)
chunks = splitter.split_documents(docs)

For code, swap the separator list for language-specific tokens:

from langchain.text_splitter import Language

code_splitter = RecursiveCharacterTextSplitter.from_language(
    language=Language.PYTHON,
    chunk_size=1500,
    chunk_overlap=150,
)

This keeps function and class boundaries intact where possible. The practical win over fixed-size is meaningful: internal benchmarks from LlamaIndex (2023) showed a ~10% improvement in answer faithfulness on technical docs just from switching to recursive over character-based splitting.

Instead of splitting on syntax, semantic chunking embeds every sentence, then looks for cosine-similarity drops between adjacent sentences. A big drop means a topic shift — that's where you cut.

from langchain_experimental.text_splitter import SemanticChunker
from langchain_openai import OpenAIEmbeddings

chunker = SemanticChunker(
    OpenAIEmbeddings(model="text-embedding-3-small"),
    breakpoint_threshold_type="percentile",   # or "standard_deviation", "interquartile"
    breakpoint_threshold_amount=95,           # cut at similarities below the 95th-percentile drop
)
chunks = chunker.create_documents([text])

Three threshold strategies:

Cost: Every sentence gets embedded at index time, roughly 3-5x the API cost of fixed-size. Chunk count is non-deterministic, which complicates infrastructure planning. Worth it for heterogeneous documents like legal briefs or research papers where topic shifts are meaningful signals.

A single chunk size forces a trade-off between precision (small chunks) and context (large chunks). Hierarchical chunking sidesteps it by building two layers: small child chunks for retrieval, large parent chunks for generation.

The retrieval flow:

1. Embed and search the small child chunks (high precision)
2. When a child chunk is a hit, fetch its parent chunk instead
3. Pass the full parent to the LLM (rich context)

from langchain.retrievers import ParentDocumentRetriever
from langchain.storage import InMemoryStore
from langchain.vectorstores import Chroma
from langchain.text_splitter import RecursiveCharacterTextSplitter

child_splitter = RecursiveCharacterTextSplitter(chunk_size=400)
parent_splitter = RecursiveCharacterTextSplitter(chunk_size=2000)

retriever = ParentDocumentRetriever(
    vectorstore=Chroma(embedding_function=embeddings),
    docstore=InMemoryStore(),
    child_splitter=child_splitter,
    parent_splitter=parent_splitter,
)
retriever.add_documents(docs)

A variant is summary-then-detail: embed an LLM-generated summary of each section as the search target, return the original section as context. This costs more at index time but retrieves the right section even when the query uses vocabulary absent from the raw text.

Late chunking flips the usual order. Instead of chunking the document and then embedding each chunk independently, you pass the entire document through the embedding model in one forward pass, collecting per-token contextual embeddings. Only after that do you pool tokens into chunk vectors.

This preserves cross-chunk context during embedding — a pronoun at token 600 carries information from its antecedent at token 100 because they shared the same attention computation.

# Pseudo-code for late chunking with a long-context encoder (e.g., jina-embeddings-v3)
import numpy as np

def late_chunk_embed(text: str, chunk_spans: list[tuple[int,int]], model) -> list[np.ndarray]:
    token_embeddings = model.encode_tokens(text)   # shape: (seq_len, dim)
    chunk_vectors = []
    for start, end in chunk_spans:
        pooled = token_embeddings[start:end].mean(axis=0)   # mean-pool span
        chunk_vectors.append(pooled / np.linalg.norm(pooled))
    return chunk_vectors

The catch: your encoder needs a context window large enough to hold the document. Jina AI's jina-embeddings-v3 (8 192 tokens) and nomic-embed-text-v1.5 (8 192 tokens) are practical options. For book-length documents you still need to pre-split into sections before applying late chunking within each section.

Jina AI published a benchmark (2024) showing late chunking outperformed fixed-size by 5–12 percentage points on BEIR retrieval tasks.

No single strategy wins everywhere. Here's how to pick:

Chunk size heuristics (for embedding models with 512-token input limit):

• Small retrieval chunks: 256–400 tokens
• Parent/generation chunks: 1 500–2 500 tokens
• Overlap: 10–15% of chunk size

Heads up: Many embedding models silently truncate input beyond their maximum. A 2 000-token chunk fed to a 512-token model embeds only the first 512 tokens — the rest is invisible. Always verify model.max_seq_length before setting chunk size.

A chunk without metadata is a dead end. At minimum, store:

{
  "chunk_id": "doc-42-chunk-7",
  "doc_id": "doc-42",
  "source_url": "https://docs.example.com/api/auth",
  "section_title": "OAuth 2.0 Flow",
  "page": 3,
  "token_count": 387,
  "created_at": "2025-11-01T09:14:00Z"
}

This enables:

• Source attribution — tell users which document answered their question
• Metadata pre-filtering — restrict vector search to chunks from a specific doc or date range before scoring
• Deduplication — detect and skip near-identical chunks from re-ingested documents

Chunk hygiene rules that move the needle in practice:

• Strip boilerplate headers/footers before chunking (page numbers, nav bars, cookie banners)
• Normalize whitespace — double spaces and \r\n cause separator mismatches
• For PDFs, run a PDF-to-Markdown converter (pymupdf4llm, marker) rather than raw text extraction — table structure is preserved and column-ordering artifacts are fixed

The only reliable signal is retrieval quality on a held-out question set. Build one early.

RAGAS metrics are the standard toolkit:

from ragas import evaluate
from ragas.metrics import context_precision, context_recall, faithfulness

result = evaluate(
    dataset=qa_dataset,           # HuggingFace Dataset with question / contexts / ground_truth
    metrics=[context_precision, context_recall, faithfulness],
)
print(result.to_pandas())

• Context precision — of the retrieved chunks, what fraction are actually relevant?
• Context recall — of the ground-truth answer's supporting facts, how many were covered by the retrieved chunks?
• Faithfulness — does the LLM's answer stick to what the chunks say, or hallucinate?

A chunking change that improves context recall without hurting precision is almost always a net win. Run ablations: same corpus, same embedding model, same retriever — only change the chunking strategy. Typical evaluation set size: 100–300 questions is enough to detect a 5% difference with statistical significance.

A few things that bite teams after the prototype stage:

Re-chunking on schema change. If you decide to change chunk size or strategy, you must re-embed and re-index the entire corpus. Plan your vector store schema (Pinecone namespace, Qdrant collection, pgvector table) to allow a blue-green index swap with zero downtime.

Multimodal documents. PDFs with figures and tables don't chunk well as plain text. Consider storing figure descriptions generated by a vision model as separate chunks with a content_type: figure metadata tag, then route queries that need visual context to those chunks.

Chunk size vs. LLM context window. With 128 k-token LLM context windows now common, there's a temptation to just pass everything in. This works for single-document Q&A but fails at scale: retrieving 50 large parent chunks × 2 000 tokens = 100 000 tokens per request. Cost and latency compound quickly.

Incremental ingestion. Appending new documents to an existing index is fine. But if a source document is updated, you must detect which chunks changed (hash the source), delete stale chunks by doc_id, and re-index only the updated sections — not the whole corpus.