Chunking strategies for RAG: German and structured documents

Making German technical documents reliably searchable? WZ-IT builds RAG systems on your own infrastructure, including document preparation, chunking, metadata and test questions from your collection. The RAG Proof of Value evaluates a scoped knowledge collection against agreed reference questions and verifiable sources. See the RAG Proof of Value · Internal AI assistants

A RAG system does not retrieve documents, it retrieves passages. How those passages are cut helps decide whether the answer contains the right source or half a rule without its condition. Most guides recommend a fixed size of a few hundred tokens and base that on English sample texts. German documents, especially from public administration, law and engineering, have different requirements: compound words, statutory structure, abbreviations, tables and scanned legacy records. This article sets out the strategies and shows what matters for German texts. As of September 2026.

Contents

• What chunking decides in a RAG system
• What makes German documents different
• The strategies at a glance
• Parent-child and overlap
• Chunk size and embedding model input limits
• Metadata every chunk should carry
• Tools and their defaults
• Test chunking instead of guessing

What chunking decides in a RAG system

During indexing, every document is extracted and split into chunks, and every chunk receives an embedding. For each query, the system looks up the most similar chunks and hands them to the language model as context. How RAG works overall is explained in What is RAG?.

This creates two opposing requirements:

• Small chunks match more precisely. An embedding condenses a chunk's content into one vector; the more topics it contains, the less distinct the vector becomes.
• Large chunks carry more context. A sentence such as "It must be inspected annually" is worthless without the paragraph before it.

Chunking is therefore a design decision with a direct effect on retrieval quality, citations and answer length.

What makes German documents different

Compound words and tokenizers

Tokenizers split text into pieces from a fixed vocabulary. English words are often contained as whole units, German compounds rarely. Our own count (September 2026) shows this for the word "Grundstücksverkehrsgenehmigung" (a land transaction permit):

Two consequences follow. German text uses more tokens than English for the same content, so a chunk's token budget holds less content. The scale of this effect across languages is described by Petrov et al. (NeurIPS 2023). And the standalone word "Genehmigung" (permit) is a single token in the BGE-M3 vocabulary, while inside the compound it is spread across "ge · nehm · igung". A search for "Genehmigung" therefore does not reliably hit the compound through exact term matching. For keyword search, decompounding filters such as the dictionary decompounder in OpenSearch solve this; how keyword and vector search are combined is covered in Hybrid search and reranking.

Long sentences and abbreviations

German technical and legal texts contain long sentences with embedded clauses and many abbreviations ending in a full stop: Abs. (paragraph), Nr. (number), gem. (pursuant to), z. B. (for example), i. V. m. (in conjunction with). Sentence splitters that treat a full stop as a sentence end cut at these points. The untrained NLTK PunktSentenceTokenizer, which the LlamaIndex SentenceSplitter uses for sentence boundaries, splits the sample sentence "Die Frist nach § 4 Abs. 2 Satz 1 beträgt drei Monate." (the deadline under section 4(2), first sentence, is three months) into "Die Frist nach § 4 Abs." and "2 Satz 1 beträgt drei Monate." (our own test, September 2026). If such a boundary falls between two chunks, the reference is incomplete in both. The remedies are a German sentence splitter, an abbreviation list, or a method that splits at paragraphs and headings rather than sentences.

Structure: sections, subsections, numbers

Statutes, bylaws, administrative regulations and procedural instructions are structured hierarchically: part, chapter, section (§), subsection (Absatz), sentence, number. A subsection such as "(2) Notwithstanding subsection 1 ..." only makes sense together with its section. Two rules help:

• Align boundaries with the structure. A chunk starts at a section or subsection, not in the middle of one.
• Carry the structural path. Every chunk gets the document title and path (for example "Bylaws, § 4 Fees, subsection 2") as metadata or as prepended context. That keeps a hit citable and unambiguous for the embedding.

Tables

Fee schedules, data sheets, inspection plans and deadline overviews hold their key statements in tables. When a character-based splitter cuts a table, rows end up in a chunk without their column headers, and the number "30" loses its meaning. Tables should therefore be handled as units of their own: complete if they are small enough, otherwise row by row with a repeated header.

Scanned PDFs and legacy records

Many collections exist only as scans. Text recognition then produces errors that chunking amplifies: hyphenation at line ends ("Verwal-" / "tung"), headers and footers in the middle of running text, columns in the wrong reading order. Before chunking, hyphenated words should be rejoined and page furniture removed. For historical documents in Fraktur (blackletter) type, Tesseract provides dedicated models (deu_latf and the Fraktur script model in tessdata_best). How documents are prepared reliably is covered in Process documents with AI.

The strategies at a glance

Recursive is the common default. LangChain's RecursiveCharacterTextSplitter tries the separators blank line, line break, space and finally single characters by default.

Structure-based is usually the better first choice for structured German documents. If the text is available as Markdown, the MarkdownHeaderTextSplitter splits at headings and stores the heading hierarchy as metadata. For PDFs and Office documents, Docling and Unstructured take care of structure detection (see below). The section sign and subsection numbers can additionally be defined as split patterns.

Semantic sounds like the obvious improvement, but it is not necessarily one. Qu, Tu and Bao (2024) conclude that the computational cost of semantic chunking is not justified by consistent performance gains. Semantic chunking is worth testing for long texts without headings, such as minutes or expert reports written as running text.

Late chunking was described by Günther et al. (Jina AI). The embedding model first processes the whole text (or the largest part that fits its context window) and produces a vector for every token. Only then is the token sequence split at the chunk boundaries and averaged per chunk. A sentence such as "It must be inspected annually" thus receives a vector that carries the installation named earlier. The method needs no additional training. Two limitations: chunk boundaries still have to be defined, and the serving layer has to return token vectors before pooling. Common embedding endpoints return only one finished vector per input; self-hosted setups need custom code for this, as shown in Jina's reference repository.

A related approach is contextual retrieval, in which a language model prepends a short context text to each chunk before indexing. It works independently of the embedding model and is described in Contextual retrieval.

Parent-child and overlap

Parent-child separates what is searched from what the language model receives. Retrieval runs over small chunks (for example single subsections), and the model receives the parent passage (for example the whole section). LlamaIndex implements this with the HierarchicalNodeParser, which by default creates three levels of 2,048, 512 and 128 tokens (LlamaIndex node parsers). For German legal and administrative texts, the pattern fits the structure well: hits at subsection level, answers with the whole section.

Overlap repeats the end of one chunk at the start of the next. It softens cuts at unfortunate points but also duplicates content in the index and in the results. About 10 to 20 percent is a common first test variant for fixed-length or recursive splitting. With structure-based boundaries, overlap is often unnecessary because the cuts already fall at natural points.

Chunk size and embedding model input limits

The upper limit is set by the embedding model: anything beyond it is truncated without an error message. The limits vary widely (as of September 2026, according to the model cards):

Three practical points:

• Measure with the right tokenizer. Chunk size has to be checked with the embedding model's tokenizer. A chunk of 500 tokens under cl100k_base can have considerably more or fewer tokens under another tokenizer.
• Context counts too. Prepending a structural path, document title or generated context text uses up token budget. With multilingual-e5, the passage: prefix is added on top.
• Long input is not a goal. A model that accepts 8,192 tokens does not mean chunks of that size are retrieved well. A few hundred to about 1,000 tokens is a common starting range, which tests with real questions confirm or shift.

Which embedding models suit German texts is compared in Best embedding models for German.

Metadata every chunk should carry

A chunk without provenance is useless in a business system: it cannot be cited, filtered or deleted selectively. These fields have proven useful:

Vector databases such as Qdrant store these fields as payload and filter directly during the search. A comparison of the options is in Qdrant vs. pgvector.

Tools and their defaults

The defaults of the common libraries differ on one important point: whether they count characters or tokens (as of September 2026, according to documentation and source code):

Docling (MIT license) converts PDF, Office files and images into a structured document model with headings, reading order and table structure, and includes text recognition, among others via EasyOCR, RapidOCR and Tesseract. The HierarchicalChunker creates one chunk per detected document element and attaches headings and captions as metadata. The HybridChunker builds on it: it splits oversized chunks according to the embedding model's tokenizer and merges undersized neighbouring chunks. For tables that span several chunks, it repeats the header row by default.

Unstructured first breaks documents into elements (title, paragraph, list, table). The by_title strategy starts a new chunk at every detected heading, even if the previous chunk still has room. Tables are always isolated and never combined with other elements; oversized tables are split into several TableChunk elements.

LangChain and LlamaIndex provide the splitters seen in most examples. Their defaults target English texts and general use. For German collections it pays to set split patterns, sentence splitter and length measurement explicitly rather than accept the defaults.

Test chunking instead of guessing

Which strategy fits only becomes clear in testing:

1. Pick a sample. Representative documents of every document type, including scans and tables.
2. Check the extraction. Look at the extracted texts before comparing chunking variants. Errors in reading order or hyphenation cannot be fixed by chunking.
3. Collect test questions. Real questions from the business with the expected source location, including questions about identifiers, table values and cross-references.
4. Compare variants. Index two or three strategies or sizes with the same embedding model and measure whether the expected source appears among the top hits.
5. Document the decision. Record strategy, parameters, tokenizer and chunking version so later changes remain comparable.

Which metrics are suitable and how a test set is built is covered in RAG evaluation.

What this means for your project

The right chunking strategy depends more on the collection than on the tool. Well-structured documents with a clean text layer benefit from structure-based chunking with a structural path and parent-child retrieval. Scanned legacy records first need reliable extraction. Semantic chunking and late chunking are additions that pay off only when tests show a need.

The overall architecture is explained in What is RAG?. How keyword and vector search work together is shown in Hybrid search and reranking, and how chunks receive context before indexing in Contextual retrieval. Where documents come from and how changes are synchronised is described in RAG data sources: Nextcloud, SharePoint and DMS. How a public authority approaches the whole path is shown in RAG knowledge base for the public sector. Such systems run locally on the AI Cube or on a managed GPU server from WZ-IT.

Sources

• Günther et al.: Late Chunking, Contextual Chunk Embeddings Using Long-Context Embedding Models
• Jina AI: Late Chunking in Long-Context Embedding Models
• Jina AI: late chunking reference implementation
• Qu, Tu, Bao: Is Semantic Chunking Worth the Computational Cost?
• Petrov et al.: Language Model Tokenizers Introduce Unfairness Between Languages
• Docling: Chunking
• Unstructured: Chunking
• LangChain: RecursiveCharacterTextSplitter
• LangChain: MarkdownHeaderTextSplitter
• LlamaIndex: node parser modules
• OpenSearch: dictionary decompounder
• Tesseract: tessdata_best