Text chunker for RAG: chunk size and overlap

Retrieval-augmented generation (RAG) works on chunks: each piece of a document is embedded on its own and pasted into the prompt when it matches a question. So the way text is cut decides what a search can find. Most tutorials set chunk_size and chunk_overlap on a LangChain splitter and move on, but the result is hard to picture, and many online chunkers only approximate it. This tool runs a line-by-line port of LangChain’s splitters (langchain-text-splitters 1.1.3 for Python and @langchain/textsplitters 1.0.2 for JavaScript). On 2026-10-12 we ran both libraries on 840 combinations of text and settings, and the tests check that every chunk here matches their output character for character.

How the recursive splitter cuts

RecursiveCharacterTextSplitter tries the separators "\n\n", "\n", " " and "" in order. It splits on the first one the text contains, keeps every piece shorter than chunk_size (strictly shorter), and splits any longer piece again with the next separator. It then glues neighbouring short pieces back together until the next one would push the chunk past the limit, and trims the whitespace from both ends.

Overlap works on those pieces. When a chunk is full, the splitter keeps its last few pieces, as many as fit within chunk_overlap, as the start of the next chunk. Pieces are never cut to fit, so the real overlap is often smaller than the setting, and zero when the final paragraph alone is longer. Pieces that had to be split again are merged on their own, so chunks either side of them never share text. With the tool’s starting settings (500 characters, 100 overlap), the sample help page splits into 4 chunks, and 3 of them start with repeated text.

Characters or tokens

By default chunk_size counts characters. Python’s len() counts Unicode code points, while JavaScript’s .length counts UTF-16 units, so an emoji is 1 character in Python and 2 in LangChain.js, and LangChain.js can cut an emoji in half when it falls back to single characters. Embedding models limit and bill in tokens, so token counting is usually what you want: in Python, build the splitter with from_tiktoken_encoder and name the encoding, because the default is the old gpt2 one. This tool offers cl100k_base, which OpenAI says to use for its text-embedding-3 models, and o200k_base. In token mode, LangChain adds up the token counts of the pieces it joins, so a chunk can come out a token or two over the limit.

Fixed token windows

TokenTextSplitter ignores structure. It encodes the whole text and takes windows of chunk_size token ids, each starting chunk_size − chunk_overlap ids after the last. Sizes are predictable, but a window can end in the middle of a word or of a character that takes several bytes, which decodes to the replacement character �, just as tiktoken does.

Markdown

The Markdown mode tries headings, code fences and horizontal rules before paragraphs. The libraries differ here: Python’s from_language(Language.MARKDOWN) uses regular expressions for headings 1 to 6, LangChain.js uses plain strings for headings 2 to 6, and Python’s MarkdownTextSplitter() class matches its patterns as literal text, so it never splits on a heading. On the sample, Python’s Markdown mode gives 4 chunks and LangChain.js’s 4, with different boundaries.

Python and LangChain.js defaults

Python defaults to 4,000 characters with 200 overlap, LangChain.js to 1,000 with 200. Python accepts an overlap equal to the chunk size; LangChain.js throws an error. Only Python can keep separators at the end of a piece instead of the start.

What it doesn’t show

Embedding cost isn’t shown, because our daily price data covers chat models, not embedding models. “Tokens to embed” is the number to multiply by your embedding model’s price; the RAG chunk size guide lists dated prices and works through the sums. If your text contains special-token markers such as <|endoftext|>, LangChain’s tiktoken helpers raise an error by default; this tool counts them as plain text. To see how many retrieved chunks fit a model, use the context window checker.