Document Chunking

Document chunking is a preprocessing step that turns documents into retrievable passages. A typical RAG ingestion pipeline extracts text, removes unwanted formatting, divides the text into chunks, attaches metadata, generates embeddings, and writes the results to a searchable index. At query time, the application retrieves relevant chunks and supplies their text to a language model. Chunking is not summarization: it generally partitions source content rather than replacing it with a shorter interpretation.

Fixed-size chunking divides text using a token or character limit, often with overlap between adjacent chunks. Token-based limits are useful because embedding and generation models enforce token budgets, although different models can use different tokenizers. Structure-aware chunking follows headings, paragraphs, sentences, or document sections. Semantic chunking identifies boundaries based on changes in meaning. Hierarchical chunking creates smaller child passages within larger parent sections so retrieval can locate precise evidence while returning broader context.

For example, a maintenance manual might describe an error code in one paragraph and its required safety precautions in the next. Splitting exactly between those paragraphs could retrieve the repair procedure without the warning. Preserving the section or adding controlled overlap can keep the instructions together. Tables require similar care: a row of values without its column headers or units may be misleading. Scanned documents often need optical character recognition and layout extraction before useful chunk boundaries can be selected.

Small chunks can isolate relevant facts, but they may lose definitions, qualifications, or relationships that span passages. Large chunks preserve more context but can dilute embedding relevance and consume more of the generation model's context window. Overlap reduces boundary-related information loss, yet increases embedded text volume, storage, and potentially redundant retrieval results. There is no universally correct chunk size or overlap percentage. A starting configuration such as 500-token chunks with 50-token overlap is an experiment, not a standard.

Metadata should preserve provenance and useful retrieval constraints. Common fields include the source URI, document identifier, title, heading path, page number where available, version, timestamp, language, and access-control labels. Document-level metadata can be inherited by chunks, while section or page metadata may need extraction during parsing. Metadata filters can restrict results to the correct product version or tenant, but sensitive-content access must be enforced through trusted authorization logic rather than relying on the model to ignore unauthorized passages.

In a cloud implementation, object storage can hold source files while managed ingestion or custom compute performs parsing and chunking. A vector database stores chunk embeddings together with text and metadata, or references to that text. Configure the vector field dimension to match the embedding model and select supported indexing and similarity settings appropriate to that model. Chunk size and embedding dimensionality are separate decisions: changing the number of tokens in a chunk does not change the model's output vector dimension.

Amazon Bedrock Knowledge Bases supports fixed-size, semantic, hierarchical, and no-chunking configurations for supported ingestion paths. No chunking treats each document as one chunk; it does not remove embedding-model input limits. Select the strategy when configuring the data source, because its chunking configuration cannot subsequently be changed in place. A different strategy requires a new data source and re-ingestion. With hierarchical retrieval, replacing matching child chunks with their parents can produce fewer returned results than the requested retrieval count.

Evaluate chunking with representative questions and known supporting passages. Measure evidence recall, answer correctness, citation quality, latency, and cost while varying chunk boundaries, overlap, retrieval count, metadata filters, and reranking. Retrieving more chunks does not automatically improve answers: duplicates and irrelevant context can crowd out useful evidence. Version the parsing and chunking configuration, and reprocess affected documents when it changes so the index remains consistent with the intended retrieval behavior.