Deduplication

Deduplication is a data-quality operation that identifies redundant records, events, or documents and retains an appropriate representation. In cloud pipelines, duplicates commonly result from overlapping source systems, repeated file uploads, connector retries, and at-least-once message delivery. Without deduplication, an integrated sales dataset can overstate revenue, while a retrieval-augmented generation application can return several copies of one document instead of diverse supporting evidence.

Begin by defining what makes two items duplicates. Exact matching can compare complete records, selected business keys, or hashes of canonical content. A source identifier combined with an event identifier is often suitable for detecting replayed events. When integrating multiple sources, source-local identifiers alone usually cannot identify the same business entity across systems. For example, customer 123 in a billing system might be customer 987 in a support platform. Cross-source matching therefore requires a shared identifier, an identifier mapping, or carefully validated entity-matching rules.

Normalize data before comparing it, but only where the transformation preserves business meaning. Standardizing telephone numbers, Unicode representation, whitespace, and timestamp formats can expose duplicates that raw string comparisons miss. Hashing normalized content supports efficient exact-match lookup, but hashes do not recognize paraphrases. Exclude ingestion timestamps and other volatile metadata from a content fingerprint when they should not distinguish copies. Conversely, retain fields such as document version, transaction type, or tenant identifier when they define meaningful boundaries.

In a batch pipeline, combine source records into a common schema and group them by the chosen duplicate key. Use a deterministic rule to select the surviving record, such as trusted source priority followed by the latest valid business timestamp and a stable tie-breaker. SQL DISTINCT removes identical projected rows; it does not resolve conflicting attributes or select the latest record per business key. A window function such as ROW_NUMBER can implement that selection. Preserve source lineage and rejected-record references so engineers can explain and reverse incorrect consolidation.

Streaming deduplication requires state. A consumer can remember processed event identifiers in a state store or use conditional writes to claim an identifier before processing. Bound that state with an explicit retention period based on retry and replay requirements. Duplicates arriving after the retention period can be processed again. A separate marker write and downstream side effect are not automatically atomic: a crash between them can either lose work or repeat it. Use transactions where supported, idempotent destination writes, or a recoverable processing protocol. Deduplication is not an automatic end-to-end exactly-once guarantee.

For foundation model consumption, deduplicate documents before chunking and embedding where possible, then consider chunk-level checks for repeated passages across otherwise different documents. Repeated content can crowd retrieval results, waste context tokens, and overrepresent one source during training or evaluation. Near-duplicate detection can use text fingerprints or embedding similarity, but similar meaning does not prove equivalence. Two policy documents may differ only in an effective date or a critical exception. Keep versions and provenance, and evaluate matching thresholds against labeled examples before deleting content.

Text reformatting with Amazon Bedrock, entity extraction with Amazon Comprehend, or normalization in AWS Lambda can prepare inputs for matching; none inherently proves that two records are duplicates. Model-generated transformations can also introduce errors, so preserve originals and validate outputs. Measure duplicate rates, missed duplicates, and false merges alongside runtime and state-storage costs. Deduplication is not compression, which changes storage representation, nor is it a substitute for validation, conflict resolution, access control, or correct source-system semantics.