Text Chunking

Text chunking is the division of text into smaller units that can be processed independently or combined selectively. It is commonly used during document ingestion for retrieval-augmented generation, or RAG, and when processing documents that exceed a foundation model's input capacity. Chunking is not the same as tokenization: tokenization converts text into model-specific tokens, while chunking groups text into useful processing units.

In a cloud-based RAG pipeline, documents are extracted, cleaned, divided into chunks, embedded, and stored in a searchable index. Each chunk should retain metadata such as its source identifier, document version, section heading, page number, and access-control attributes. At query time, the application retrieves relevant chunks and places selected evidence into the generation prompt. Stable identifiers and source offsets make it possible to diagnose missing passages and generate traceable citations.

Chunk boundaries should reflect both document structure and model constraints. Fixed-size splitting is predictable but can separate a definition from its qualifications. Structure-aware splitting respects paragraphs, headings, code blocks, and tables. Semantic splitting attempts to detect topic changes, while hierarchical approaches maintain relationships between small searchable passages and larger sections. For example, a support manual might use individual troubleshooting procedures as chunks rather than splitting every page at an arbitrary character count.

Chunk size creates a practical trade-off. Small chunks can improve retrieval precision but may omit essential context; large chunks can include useful explanations but dilute the embedding with unrelated topics. Overlap can preserve information near boundaries, but increases embedding cost, index size, and duplicate retrieval. A starting experiment might compare 400-token and 800-token chunks with modest overlap, but no size or overlap percentage is universally correct. Evaluate representative questions and document types rather than adopting a fixed setting without testing.

Token budgeting must consider the complete model request, not just retrieved text. Account for system instructions, conversation history, user input, tool definitions where applicable, formatting overhead, and the model's output allowance under its documented limits. Embedding models also have their own input limits and tokenizers. Character counts are only approximations, especially for multilingual text or code. Dynamic chunking can adapt boundaries and sizes to document structure and current processing budgets; query-time context packing separately determines which retrieved chunks fit in a generation request.

When information appears to be missing, distinguish explicit context-window errors from silent truncation introduced by application code, tokenizers, connectors, or model configuration. Record token counts, chunk counts, source coverage, ingestion failures, assembled prompt sizes, and relevant finish reasons without exposing sensitive content. Check whether the missing passage survived extraction, appeared in a chunk, reached the index, was retrieved, and remained in the final prompt. An output stopped by a generation-token limit is a different problem from omitted input evidence.

Chunking a long document does not ensure that every part is processed. For an exhaustive review, track all chunks through processing, retry failures, and reconcile results across boundaries. A contract review may require separate checks for obligations, exceptions, and definitions, followed by a consolidation step that links their source passages. Hierarchical summarization can reduce context requirements, but summaries are lossy and may remove details needed later. Preserve original chunks so important claims can be checked against source text.

Retrieval troubleshooting should separate chunking defects from extraction, embedding, and search defects. Inspect retrieved passages for representative queries and measure whether they contain the evidence required to answer. Check malformed text, repeated headers, broken tables, missing headings, and inconsistent preprocessing between ingestion and querying. Verify embedding-model compatibility, vector dimensions, similarity configuration, and model-specific normalization requirements. A valid vector does not prove that the source content was complete or semantically useful.

Monitor retrieval quality as document layouts, languages, query patterns, or embedding models change. Compare evidence recall, ranking quality, answer relevance, groundedness, latency, and cost against a labeled evaluation set. Changing chunk boundaries requires regenerating affected chunks and embeddings and removing stale entries; changing embedding models usually requires re-embedding indexed content. If retrieval finds the correct evidence but answers remain poor, investigate prompt organization, conflicting context, duplication, and generation behavior instead of assuming that chunk size is always the cause.