Choosing Token Chunk Size for RAG Pipelines

Choosing Token Chunk Size for RAG Pipelines: Token Counter Guide

When a chunker splits by words, a RAG prompt can overflow before retrieval even starts.

A document chunker that splits every 500 words assumes tokens track words one-to-one. They do not: a 500-word technical document with inline code, URLs, and acronyms may produce 700 or more tokens, not 500. OpenAI's English rule of thumb is 100 tokens per 75 words, so dense technical prose can easily run above a word-count estimate.¹ When that gap accumulates across four retrieved chunks, your context window may overflow without an obvious error. Counting tokens at chunk design time, not word count, is the one habit that prevents these invisible overflows.

How chunk token counts affect context window usage

A typical RAG prompt includes a system instruction, retrieved context chunks, and a user query, and every one of these components competes for space within the same finite context window budget. LlamaIndex's end-to-end RAG example assembles documents into text chunks, embeds each node, stores the nodes, and then retrieves the most relevant nodes for a query before sending them to a query engine for final answer generation.² If the system instruction is 800 tokens and the user query is 200 tokens, a 128,000-token window leaves 127,000 tokens for retrieved chunks, and that budget must also accommodate the model's response.

Dividing the remaining context budget by your chosen chunk size gives the maximum number of chunks you can include in a single request. Consequently, chunk size is a direct control knob on retrieval breadth: smaller chunks fit more sources within the same budget, while larger chunks provide more coherent passages per source at the cost of fewer total chunks.

Allocating a RAG prompt budget before retrieval

Always reserve tokens for the system prompt, user query, and expected response before choosing the chunk count for your pipeline. A 300-token chunk is not just a retrieval setting; it determines how many sources can fit once the fixed prompt overhead is included. On a 1M-token model with a 1,500-token system prompt, a 200-token user query, and a 5,000-token response reserve, the remaining 993,300 tokens can fit 3,311 chunks at 300 tokens each, but most RAG pipelines retrieve only 5 to 20 chunks per query, which means the chunk size setting has a larger impact on per-chunk context quality than on the total number of chunks that can fit.

Picking chunk size by content type

Code and prose tokenize very differently, and the right chunk size varies significantly by content type in ways that directly affect both retrieval quality and context window efficiency. Prose paragraphs of 150-250 tokens (roughly 120-200 words) tend to preserve semantic coherence while leaving room for many chunks within a single request.

For technical documentation with code examples, 200-350 tokens per chunk accounts for the higher token density of code, where operators and brackets inflate the count well beyond what a word-length estimate would suggest. For legal or regulatory text, which uses long sentences and specialized vocabulary, 300-500 tokens often captures enough context for an answer without fragmenting key clauses across chunk boundaries. Pinecone frames the same trade-off as finding chunks large enough to hold meaningful information and small enough for performant retrieval, then warns that chunks that are too small or too large can reduce search precision.³

Matching RAG chunk size to document type

Measure a few real chunks from each document type before choosing one splitter setting for the entire corpus. Legal documents with long, complex sentences and cross-references between clauses benefit from larger 400-to-500-token chunks that keep related provisions together, while technical API documentation with discrete function descriptions works better with smaller 150-to-250-token chunks that isolate each function's behavior; choosing a single chunk size for a mixed corpus without measuring the token density of each document type leads to either fragmented legal clauses or bloated code sections that waste context window space.

Testing and calibrating chunk sizes

Practical calibration requires measuring actual chunk token counts on your specific corpus rather than relying on theoretical estimates that may not reflect your content's true token density. Paste a representative sample from each document type in your corpus into this tool and note the token count per paragraph or section to build an empirical understanding of how your content tokenizes.

Verifying the assembled prompt fits before each request

From those measurements, calculate the average chunk size your splitter will produce for each content type and verify that N chunks plus system prompt plus query fits your target model's context window with adequate room for the response. LangChain's RecursiveCharacterTextSplitter exposes chunk_size and chunk_overlap as explicit configuration parameters, with chunk_size measured by its length_function, so token-based splitting requires a tokenizer-backed length function rather than a simple character count.⁴ Furthermore, test retrieval quality with different chunk sizes before committing to a production setting, because semantic coherence and retrieval precision often improve when chunk boundaries align with natural document structure rather than arbitrary character or token limits.

Keep the chunk size as a configurable value rather than a hardcoded constant, because a single splitter setting that works for one document type will silently mis-size another and push the assembled prompt over the window without any visible error. Revalidating the fit after any change to the splitter, the overlap, or the retrieval count is what stops slow context growth from surprising you at request time.

Balancing chunk size and conversation history in multi-turn RAG

Balancing chunk size and conversation history in multi-turn RAG conversations requires a dynamic token budget because the history grows with each turn. A single-turn RAG prompt can allocate the full context budget to retrieved chunks. A ten-turn conversation must share that budget between chunks and accumulated history, and the history share grows with every exchange. On a 128,000-token context window with a 1,000-token system prompt, roughly 127,000 tokens remain for both retrieved content and conversation history combined.

A starting allocation is 40% of available context for conversation history and 60% for retrieved chunks. At 127,000 tokens of available context, that reserves about 50,800 tokens for history and 76,200 for chunks. At 300 tokens per chunk, you can include 254 chunks per turn. As the conversation grows, reduce chunk count proportionally to maintain the split. Implement this as a dynamic calculation at assembly time, not a fixed constant, so the allocation adjusts naturally as the conversation lengthens.

Embedding model token limits and LLM chunk size compatibility

Across most RAG architectures, the embedding model and the LLM enforce separate and independent token limits that can conflict. OpenAI's third-generation embedding models list a max input of 8,192 tokens, while Sentence Transformers notes that BERT-style embedding models commonly cap at 512 tokens and truncate longer texts to model.max_seq_length.⁵⁶ Chunks that fit within the embedding model's limit encode correctly; chunks that exceed it are silently truncated or raise an error depending on the library and its configuration.

Your chunk size must satisfy both limits simultaneously: small enough for the embedding model to encode the full chunk, and large enough to contain sufficient context for the LLM to answer the query. For embedding models with 512-token limits, this constrains chunks to roughly 400 tokens to leave a safety margin. Paste a representative chunk and try measuring one chunk against both ceilings, reading its count directly against the embedding model limit and the LLM context window before you lock in a chunk_size value in your text splitter config.

When to use this

Use this when designing a new RAG pipeline or diagnosing context overflow errors in an existing one. You should paste representative document passages to measure actual token density before setting the chunk size in your text splitter.

Examples

A legal clause averages 180 words. Assumed chunk size: 180 tokens. Actual measured count: 240 tokens.

At 6 retrieved chunks per query: 6 x 240 = 1,440 tokens. With 800-token system prompt and 200-token query: 2,440 total. Well within most model limits.

A function with docstring: 80 words. Assumed chunk: 80 tokens. Actual: 140 tokens (dense symbols).

At 20 retrieved code chunks: 20 x 140 = 2,800 tokens. Adjust to 10 chunks, or reduce chunk size to 70 tokens to match the assumed budget.