# Deduplication

Deduplication identifies repeated representations of the same data and removes or consolidates them according to explicit matching and retention rules.

Cloud concept. Updated 2026-10-29. Page: https://blowstack.com/tools/cloud-concepts-hub/deduplication

> Duplicates arise when sources overlap, ingestion jobs replay data, producers retry delivery, or documents appear in multiple formats. Deduplication improves analytical accuracy and foundation model input quality while reducing unnecessary storage, processing, embedding, and retrieval work. Its correctness depends on distinguishing redundant copies from legitimate repeated events or meaningful versions.

Practice exams covering Deduplication: [DEA-C01 AWS Certified Data Engineer - Associate](https://blowstack.com/practice-exams/aws-certified-data-engineer-associate), [AIP-C01 AWS Certified Generative AI Developer - Professional](https://blowstack.com/practice-exams/aws-certified-generative-ai-developer-professional).

## 6 key aspects of Deduplication

Reliable deduplication combines explicit identity rules, safe normalization, deterministic retention, bounded processing state, and measurable matching quality.

1. **Explicit duplicate identity**: Define whether equality applies to an event, business entity, document, or content version. Use stable identifiers or carefully selected composite keys rather than assuming that matching names or identical payloads always indicate redundancy.
2. **Normalization before matching**: Apply consistent, meaning-preserving formatting before calculating keys or comparing records. Preserve original values and avoid transformations that erase important distinctions, such as case-sensitive identifiers or document versions.
3. **Deterministic survivor selection**: Specify which record survives and how conflicting attributes are handled. Source priority, business timestamps, and stable tie-breakers make reruns reproducible and prevent arbitrary record selection.
4. **State and replay boundaries**: Streaming pipelines must retain enough identifier state to cover expected retries and replays. Coordinate deduplication state with destination writes so a crash does not silently discard work or repeat side effects.
5. **Validated similarity matching**: Use approximate text matching or embeddings when duplicates differ in wording or formatting. Validate thresholds against representative examples because false merges can remove distinct transactions, policies, or evidence.
6. **Traceability and quality measurement**: Retain provenance, duplicate-group identifiers, and the reason each item was retained or suppressed. Monitor both missed duplicates and incorrectly merged items rather than treating a higher removal count as proof of better quality.

## Cloud services for Deduplication

A practical pipeline can use AWS Glue to integrate historical datasets and Lambda to normalize incoming events. DynamoDB can hold event identifiers or content fingerprints for conditional duplicate checks, with explicit retention and recovery rules. For document collections, Amazon Comprehend can extract matching features, while Amazon Bedrock can support text restructuring or embedding generation. OpenSearch Service can find similar-content candidates for further validation. Apply deterministic survivor rules before publishing curated data or generating final retrieval indexes, and retain original inputs and lineage so incorrect matches can be investigated and reversed.

- **[AWS Glue](https://blowstack.com/tools/cloud-services-hub/aws-glue)**: AWS Glue ETL jobs can normalize schemas, combine source datasets, and use Spark operations to remove duplicates or select survivors by business key. Job bookmarks track previously processed input but do not replace record-level deduplication across overlapping sources or replayed records.
- **[AWS Lambda](https://blowstack.com/tools/cloud-services-hub/aws-lambda)**: Lambda functions can normalize incoming records, calculate content fingerprints, and apply duplicate-checking logic before downstream processing. Functions must account for retries and coordinate with persistent state or idempotent destinations rather than relying on in-memory identifiers.
- **[Amazon DynamoDB](https://blowstack.com/tools/cloud-services-hub/amazon-dynamodb)**: DynamoDB conditional writes can atomically create a deduplication marker only when its key does not already exist. TTL supports eventual cleanup, but applications must explicitly evaluate expiration when immediate reuse is required because expired items are not deleted immediately.
- **[Amazon OpenSearch Service](https://blowstack.com/tools/cloud-services-hub/amazon-opensearch-service)**: OpenSearch Service can support exact fingerprint lookup and vector similarity searches for candidate near-duplicates. Similarity results require application-level rules and validation before records are suppressed or merged.
- **[Amazon Bedrock](https://blowstack.com/tools/cloud-services-hub/amazon-bedrock)**: Models accessed through Amazon Bedrock can reformat text into a consistent structure, while embedding models can produce vectors for near-duplicate candidate detection. Neither reformatting nor embedding generation automatically performs deduplication; the pipeline must implement matching, validation, and survivor selection.
- **[Amazon Comprehend](https://blowstack.com/tools/cloud-services-hub/amazon-comprehend)**: Amazon Comprehend can extract entities that help construct matching features across differently formatted text. Entity extraction is a preparation step, not an identity guarantee, and ambiguous names require additional evidence before consolidation.

## Deduplication exam questions

Real practice exam questions on Deduplication, with the correct answer and its explanation.

### Question 1 (DEA-C01)

**Question:**

A company uses an AWS Glue Spark job to combine policy documents exported from two content repositories into an Amazon S3 dataset for a search application. Each record contains tenant_id, revision, document_text, source_record_id, and ingestion_timestamp. The repositories assign different source_record_id values to copies of the same document, and retries produce additional records with different ingestion timestamps.

The company has approved a text-normalization function that removes repository-specific formatting without changing policy meaning. Copies with the same normalized text must be indexed only once within each tenant and revision. Different tenants and different revisions must remain separate, even when their text is identical. Some revisions contain no text changes.

Which transformation meets these requirements?

**Correct answer:** Normalize document_text, combine the source datasets, and remove duplicates using tenant_id, revision, and a SHA-256 hash of the normalized text as the duplicate key. Exclude source_record_id and ingestion_timestamp from the key.

**Explanation:** The normalized-text hash identifies equivalent content across repositories and retries. Including tenant_id and revision preserves the required tenant and revision boundaries. Excluding source-local identifiers and ingestion timestamps prevents delivery metadata from distinguishing redundant copies.

### Question 2 (AIP-C01)

**Question:**

A developer is preparing instruction-response examples for fine-tuning and evaluating a supported foundation model in Amazon Bedrock. The examples come from several product-support exports. Some examples differ only in formatting, while others paraphrase the same question and approved answer. Product identifiers, numerical limits, and negation can distinguish genuinely different examples.

A pilot randomly divides the records into training and evaluation sets. Evaluation scores are unusually high because lightly edited copies of training examples appear in the evaluation set. The company requires one representative of each redundant example group across the entire dataset, preservation of materially different examples, and retention of source lineage. Reviewers can adjudicate uncertain matches before dataset publication.

Which preprocessing design best meets these requirements?

**Correct answer:** Before partitioning, identify exact matches using meaning-preserving normalization and content hashes. Use Amazon Bedrock embeddings to propose additional matching pairs, validate equivalence while checking critical distinctions, and hold uncertain matches for review. Keep one deterministic representative per validated group with all source references, and then create the training and evaluation sets.

**Explanation:** This detects both normalized exact copies and potential paraphrases without treating semantic similarity as proof of equivalence. Validation and review protect meaningful differences. Selecting representatives across the entire dataset before partitioning prevents redundant examples from leaking between training and evaluation, while retained source references preserve lineage.

## Practice Deduplication in a full exam

Questions on Deduplication in each practice exam, with every answer option and an explanation for each:

- [AIP-C01](https://blowstack.com/practice-exams/aws-certified-generative-ai-developer-professional): 3 questions
- [DEA-C01](https://blowstack.com/practice-exams/aws-certified-data-engineer-associate): 2 questions

## What is Deduplication

Deduplication is a data-quality operation that identifies redundant records, events, or documents and retains an appropriate representation. In cloud pipelines, duplicates commonly result from overlapping source systems, repeated file uploads, connector retries, and at-least-once message delivery. Without deduplication, an integrated sales dataset can overstate revenue, while a retrieval-augmented generation application can return several copies of one document instead of diverse supporting evidence.

Begin by defining what makes two items duplicates. Exact matching can compare complete records, selected business keys, or hashes of canonical content. A source identifier combined with an event identifier is often suitable for detecting replayed events. When integrating multiple sources, source-local identifiers alone usually cannot identify the same business entity across systems. For example, customer 123 in a billing system might be customer 987 in a support platform. Cross-source matching therefore requires a shared identifier, an identifier mapping, or carefully validated entity-matching rules.

Normalize data before comparing it, but only where the transformation preserves business meaning. Standardizing telephone numbers, Unicode representation, whitespace, and timestamp formats can expose duplicates that raw string comparisons miss. Hashing normalized content supports efficient exact-match lookup, but hashes do not recognize paraphrases. Exclude ingestion timestamps and other volatile metadata from a content fingerprint when they should not distinguish copies. Conversely, retain fields such as document version, transaction type, or tenant identifier when they define meaningful boundaries.

In a batch pipeline, combine source records into a common schema and group them by the chosen duplicate key. Use a deterministic rule to select the surviving record, such as trusted source priority followed by the latest valid business timestamp and a stable tie-breaker. SQL DISTINCT removes identical projected rows; it does not resolve conflicting attributes or select the latest record per business key. A window function such as ROW\_NUMBER can implement that selection. Preserve source lineage and rejected-record references so engineers can explain and reverse incorrect consolidation.

Streaming deduplication requires state. A consumer can remember processed event identifiers in a state store or use conditional writes to claim an identifier before processing. Bound that state with an explicit retention period based on retry and replay requirements. Duplicates arriving after the retention period can be processed again. A separate marker write and downstream side effect are not automatically atomic: a crash between them can either lose work or repeat it. Use transactions where supported, idempotent destination writes, or a recoverable processing protocol. Deduplication is not an automatic end-to-end exactly-once guarantee.

For foundation model consumption, deduplicate documents before chunking and embedding where possible, then consider chunk-level checks for repeated passages across otherwise different documents. Repeated content can crowd retrieval results, waste context tokens, and overrepresent one source during training or evaluation. Near-duplicate detection can use text fingerprints or embedding similarity, but similar meaning does not prove equivalence. Two policy documents may differ only in an effective date or a critical exception. Keep versions and provenance, and evaluate matching thresholds against labeled examples before deleting content.

Text reformatting with Amazon Bedrock, entity extraction with Amazon Comprehend, or normalization in AWS Lambda can prepare inputs for matching; none inherently proves that two records are duplicates. Model-generated transformations can also introduce errors, so preserve originals and validate outputs. Measure duplicate rates, missed duplicates, and false merges alongside runtime and state-storage costs. Deduplication is not compression, which changes storage representation, nor is it a substitute for validation, conflict resolution, access control, or correct source-system semantics.

## Learn Deduplication in lessons

2 short lessons on Deduplication in 2 certification courses.

- DEA-C01: Deduplication: consolidating records across sources (Premium), https://app.blowstack.com/lessons/deduplication-consolidating-records-across-sources-dea-c01
- AIP-C01: Deduplication: exact and near-duplicate matching for FM inputs (Premium), https://app.blowstack.com/lessons/deduplication-exact-and-near-duplicate-matching-for-fm-inputs-aip-c01
