- Gestion automatique des sessions
- Ciblez n’importe quelle ville parmi 195 pays
- Nombre illimité de sessions simultanées
Document Chunking
TLDR: Document chunking splits long documents into smaller segments that can be embedded, indexed, and retrieved individually. Chunk boundaries determine what a retrieval system is able to return.
Document chunking is a preparation step in retrieval systems. Embedding models accept a limited amount of text, and retrieval works best when each unit returned is coherent and self contained. A 40 page manual embedded as one vector produces a single result that is mostly irrelevant to any specific question. The same manual split into sections produces results that answer questions directly.
Why Chunk Size Matters:
- Retrieval precision: Smaller chunks match queries more exactly, because less unrelated text dilutes the embedding.
- Context completeness: Larger chunks retain surrounding information that a fragment would lose.
- Token cost: Retrieved chunks are inserted into the prompt, so chunk size directly affects cost per query.
- Answer quality: A chunk containing the answer but missing its qualifying conditions can produce a confidently wrong response.
The tension is permanent. Precision pushes toward smaller chunks, completeness pushes toward larger ones, and no single size resolves it for all content.
Chunking Strategies:
| Strategy | Splits on | Strength | Weakness |
|---|---|---|---|
| Fixed length | Character or token count | Simple, predictable sizes | Cuts mid sentence and mid idea |
| Sentence | Sentence boundaries | Never breaks a sentence | Chunks vary widely in size |
| Paragraph | Blank lines | Matches author intent | Fails on poorly formatted text |
| Structural | Headings and sections | Semantically coherent units | Requires structured input |
| Recursive | Hierarchy of separators | Degrades gracefully | More complex to configure |
| Semantic | Embedding similarity shifts | Follows topic changes | Slow and costly to compute |
Structural Chunking:
Where documents have headings, splitting on them usually outperforms size based methods. A heading marks a topic boundary that the author placed deliberately, and the text beneath it is already a coherent unit.
This is a practical reason to convert pages through HTML to Markdown before chunking. Markdown preserves heading hierarchy in a form that is trivial to split on, while raw HTML buries it in markup and plain text extraction discards it entirely.
The limitation is that many documents have no usable structure. Transcripts, scanned text, and single block pages provide nothing to split on, and those fall back to recursive or sentence based methods.
Recursive Chunking:
SEPARATORS = ["n## ", "n### ", "nn", "n", ". ", " "]
def chunk(text, max_size, separators=SEPARATORS):
if len(text) <= max_size:
return [text]
if not separators:
# No separators left: split at the size limit
return [text[i:i + max_size]
for i in range(0, len(text), max_size)]
sep, rest = separators[0], separators[1:]
parts = text.split(sep)
chunks = []
for part in parts:
if len(part) <= max_size:
chunks.append(part)
else:
chunks.extend(chunk(part, max_size, rest))
return [c for c in chunks if c.strip()]
The separator list is ordered from strongest boundary to weakest. Headings are tried first, then blank lines, then line breaks, then sentences, then individual words. Splitting only descends to a weaker boundary when the stronger one leaves a section still too large.
Overlap:
Overlap repeats a portion of each chunk at the start of the next, so that information near a boundary appears in both.
Without overlap, a sentence split across two chunks is fully present in neither, and a query matching that sentence may retrieve a fragment that makes no sense alone. With overlap, the boundary region survives intact in at least one chunk.
The cost is storage and duplication. Twenty percent overlap means twenty percent more vectors to store and search, and it means the same passage can be returned twice in a single result set, which deduplication has to handle.
Typical Starting Points:
Figures below are common starting configurations rather than measured optima. Tune against your own content and evaluation set.
| Content type | Chunk size | Overlap |
|---|---|---|
| Technical documentation | Section based | Minimal |
| Long form articles | 500 to 1000 tokens | 10 to 20 percent |
| Product descriptions | One record per chunk | None |
| Transcripts | 200 to 400 tokens | 20 to 30 percent |
| Legal and policy text | Clause based | Moderate |
Metadata and Context Preservation:
A chunk retrieved in isolation loses everything around it. Attaching metadata restores part of that context without inflating the embedded text.
Useful fields include the source URL, document title, the heading path the chunk sits under, its position in the document, and a collection timestamp. Retrieval can then present “Installation, Requirements, step 3” rather than an unattributed paragraph, and filters can restrict search to a document set or date range before ranking begins.
Some systems also prepend the heading path to the chunk text before embedding, so topical context influences the vector itself. This improves matching on short chunks at the cost of slight repetition across the index.
Chunking and Content Quality:
Chunking amplifies whatever it is given. If pages still contain navigation menus and footers, those fragments become retrievable chunks that match queries and return nothing useful. Running boilerplate removal before chunking is what prevents an index from filling with cookie banners.
Bright Data’s Crawl API returns clean Markdown with boilerplate stripped and heading structure intact, which is the input format structural chunking requires.