Hierarchical Chunking

Hierarchical chunking is a document segmentation strategy that preserves relationships between larger and smaller pieces of content. A parent chunk might represent a section of a technical manual, while its child chunks represent individual passages within that section. Retrieval can search the smaller chunks for precise matches and then provide their parent content to a foundation model. This addresses a common retrieval-augmented generation problem: a passage can match a question closely but omit information necessary to interpret the answer correctly.

For example, an operational runbook might contain a section about database failover, with separate passages covering prerequisites, execution, and recovery checks. A question about initiating failover may match the execution passage. Returning its parent section can also expose the prerequisite that replication lag must remain below a specified threshold. The hierarchy therefore supports both precise discovery and context preservation, rather than forcing one chunk size to serve both purposes.

During ingestion, a custom pipeline can parse headings, paragraphs, tables, and other structural elements before constructing parent and child records. Each child should retain a parent identifier, document identifier, source location, and document version. Embeddings commonly represent child passages, while parent text remains accessible through a document store or the retrieval system. Some implementations also index larger levels, but embedding every level is not a requirement of hierarchical chunking.

In cloud architectures, object storage can hold source documents and processed content, compute functions can implement segmentation, and a vector search service can locate relevant children. Retrieval logic then resolves parent identifiers, removes duplicate parents, and selects context within the model's token budget. Amazon Bedrock Knowledge Bases offers managed hierarchical chunking with parent and child token sizes and configurable overlap. Its retrieval behavior replaces retrieved child chunks with their parents, so the returned result count can be lower than the requested number of results.

Managed token-based hierarchy should not be confused with understanding a document's logical structure. If requirements demand that chunks follow policy clauses, Markdown headings, or complete procedures, custom processing may be necessary. AWS Lambda can implement fixed-size chunking as a baseline or execute custom hierarchy-building logic. Splitting text into equal token windows alone is not hierarchical chunking unless the implementation also establishes and uses relationships between levels.

The main trade-off is between retrieval specificity and context expansion. Small child chunks can isolate useful evidence, but excessively short passages may lack enough meaning for effective embeddings. Large parents can preserve qualifications but increase model input cost, consume context capacity, and introduce distracting material. Overlap can protect boundary-spanning statements, although it increases duplication and indexing work. Parent deduplication prevents several matching children from repeatedly adding the same section to a prompt.

Evaluate the strategy against fixed-size segmentation using representative questions, including questions that require exceptions, definitions, or adjacent procedural steps. Measure evidence retrieval, answer correctness, citation accuracy, latency, and context token consumption. Apply authorization checks to expanded parent content, not only matched children, so expansion cannot disclose restricted material. Keep related chunks consistent during updates and deletions. Hierarchical chunking is neither a guarantee of better answers nor a substitute for semantic chunking: hierarchy describes relationships between segments, while semantic segmentation describes how boundaries are chosen.