# Data Skew

Data skew is an uneven distribution of data or data-processing work across partitions, tasks, or nodes in a distributed system.

Cloud concept. Updated 2026-10-29. Page: https://blowstack.com/tools/cloud-concepts-hub/data-skew

> Data skew occurs when some partitions contain substantially more records, larger records, or more expensive processing than others. It creates bottlenecks that can cause slow transformations, memory failures, streaming backlogs, and inefficient resource use even when overall cluster capacity appears sufficient.

Practice exams covering Data Skew: [DEA-C01 AWS Certified Data Engineer - Associate](https://blowstack.com/practice-exams/aws-certified-data-engineer-associate), [DVA-C02 AWS Certified Developer - Associate](https://blowstack.com/practice-exams/aws-certified-developer-associate).

## 6 key aspects of Data Skew

Effective skew management combines partition-aware diagnosis with targeted redistribution or query changes, followed by correctness and performance validation.

1. **Uneven work distribution**: Skew can involve record counts, byte volume, or computational cost. Similar partition sizes do not guarantee similar runtimes if some records require substantially more processing.
2. **Hot keys and partitioning**: Frequently occurring values can overload the partition responsible for that key. Adding hash partitions does not divide records with the same unchanged key across multiple partitions.
3. **Task-level diagnosis**: Compare slow tasks with typical tasks using duration, shuffle bytes, spills, memory pressure, and failure patterns. Correlate these observations with key-frequency measurements rather than relying only on average cluster utilization.
4. **Targeted transformation strategies**: Broadcast joins, supported adaptive skew-join handling, partial aggregation, and selective salting address different causes of skew. Choose a strategy based on the execution plan, data sizes, and required result semantics.
5. **Limited benefit from scaling alone**: More workers help only when the workload can be divided among them. Larger workers may accommodate an oversized task, but they do not remove the distribution imbalance.
6. **Correctness and operational trade-offs**: Redistributing data can increase shuffle traffic, replication, and implementation complexity. Changes to streaming keys can also affect ordering, so test correctness alongside latency and cost.

## Cloud services for Data Skew

A pipeline might ingest events through Kinesis Data Streams, transform them with AWS Glue or Amazon EMR, and load analytical tables into Amazon Redshift. Skew can occur independently at each layer: a hot streaming partition key, an oversized Spark shuffle partition, or an uneven warehouse distribution key. CloudWatch provides operational signals, while engine-specific interfaces expose the task and distribution details needed to locate the bottleneck. Troubleshoot each boundary separately and select keys for its access and processing requirements; one partitioning strategy rarely optimizes ingestion, transformation, and analytics equally well.

- **[AWS Glue](https://blowstack.com/tools/cloud-services-hub/aws-glue)**: AWS Glue Spark jobs can encounter skew during joins, aggregations, and repartitioning. Use Spark UI information, job metrics, and logs to identify oversized tasks, then adjust transformation logic or supported Spark execution settings rather than simply increasing worker counts.
- **[Amazon EMR](https://blowstack.com/tools/cloud-services-hub/amazon-emr)**: Amazon EMR runs distributed processing engines such as Apache Spark, where hot keys can create straggler tasks and excessive spilling. Spark execution plans and task metrics help determine whether broadcast joins, salting, or version-supported adaptive skew-join optimization are appropriate.
- **[Amazon Redshift](https://blowstack.com/tools/cloud-services-hub/amazon-redshift)**: Amazon Redshift distributes table data across compute resources, so a skewed distribution key can leave some resources doing substantially more work than others. Inspect table distribution and query execution, then evaluate distribution styles or keys while considering the cost of redistributing data for joins.
- **[Amazon Kinesis Data Streams](https://blowstack.com/tools/cloud-services-hub/amazon-kinesis-data-streams)**: Kinesis Data Streams uses partition keys to map records to shards, so concentrated keys can cause hot shards and throttling. Adding shards does not spread a single unchanged partition key across shards; changing key design requires considering ordering and downstream processing requirements.
- **[Amazon CloudWatch](https://blowstack.com/tools/cloud-services-hub/amazon-cloudwatch)**: CloudWatch collects pipeline metrics and logs that help detect increasing runtime, throttling, failures, and consumer lag. Use these signals with engine-level task metrics because aggregate metrics alone may conceal a small number of overloaded partitions.

## Data Skew exam questions

Real practice exam questions on Data Skew, with the correct answer and its explanation.

### Question 1 (DEA-C01)

**Question:**

An AWS Glue Spark job joins 900 GB of clickstream records to an 8 MB reference DataFrame on campaign_id. The reference DataFrame contains one row per campaign_id and fits comfortably in each executor's available memory. Input partitions are similar in size, and reading both datasets completes quickly.

The execution plan uses a sort-merge join. During the join stage, one task reads 60% of the shuffle data and runs for 35 minutes; most other tasks finish within 2 minutes. Profiling shows that 60% of clickstream records have the same campaign_id, which has a matching reference row.

Which change will remove the join's key-based shuffle bottleneck while preserving the join results?

**Correct answer:** Change the join to a broadcast hash join that broadcasts the reference DataFrame to the executors.

**Explanation:** Broadcasting the small reference DataFrame lets executors join their local clickstream partitions without shuffling the large dataset by campaign_id. Because the input partitions are balanced and the reference data fits safely in memory, this removes the identified bottleneck while preserving the matching rows.

### Question 2 (DVA-C02)

**Question:**

A developer maintains an Amazon DynamoDB table that stores telemetry events for 100,000 tenants. The partition key is tenantId, and the sort key is eventTimestamp#eventId. One tenant produces 75% of all writes and experiences throttling during sustained traffic, even though the table has unused write capacity. Each event has a unique, randomly generated eventId. Applications retrieve a tenant's events for a specified time range and can merge results from up to 32 parallel Query requests. Which key design best distributes the busiest tenant's writes while preserving this read access pattern?

**Correct answer:** Use tenantId#bucket as the partition key, where bucket is hash(eventId) modulo 32, and retain eventTimestamp#eventId as the sort key.

**Explanation:** The varied event identifiers distribute each tenant's events across 32 derived partition key values, reducing concentration from the busiest tenant. A time-range read can query all 32 keys with sort key conditions and merge the results, staying within the application's supported fan-out.

## Practice Data Skew in a full exam

Questions on Data Skew in each practice exam, with every answer option and an explanation for each:

- [DEA-C01](https://blowstack.com/practice-exams/aws-certified-data-engineer-associate): 4 questions
- [DVA-C02](https://blowstack.com/practice-exams/aws-certified-developer-associate): 2 questions

## What is Data Skew

Distributed data processing depends on dividing work among multiple workers. Data skew undermines that parallelism when a small number of partitions receive a disproportionate share of the data or processing effort. Most tasks may finish quickly while a few overloaded tasks determine the completion time of the entire stage. In cloud pipelines, this can increase execution costs, delay downstream datasets, or cause repeated failures when overloaded workers exhaust memory.

A common cause is an uneven distribution of key values. Suppose an event dataset contains a customer\_id field, and one customer generates 40 percent of all events. A transformation that hash-partitions records by customer\_id sends that customer's records to one logical partition. Increasing the number of hash partitions does not split that single key across partitions. Null values, placeholder identifiers, and low-cardinality keys can produce similar concentrations. Unequal input sizes and differences in record-processing complexity can also create imbalanced tasks, even when record counts look balanced.

Diagnosis should begin at the slow stage rather than with cluster-wide averages. Compare task durations, input bytes, shuffle-read sizes, spill volumes, garbage-collection time, and executor failures. A few tasks with much larger shuffle reads and longer runtimes than their peers strongly suggest skew. Inspect execution plans and profile the frequency of join and aggregation keys to identify the cause. Slow tasks alone are not proof: infrastructure problems, remote-service latency, and inefficient user-defined functions can create similar symptoms.

Join strategies require particular attention. If a large fact table is joined to a genuinely small dimension table, broadcasting the small table can avoid shuffling the large table for that join. However, the broadcast data must fit safely in worker memory, and this approach does not necessarily correct preexisting input imbalance. For large joins with hot keys, supported query engines may use adaptive skew-join optimization to split oversized shuffle partitions and replicate matching data where necessary. Availability and behavior depend on the engine version and configuration.

Key salting is another technique: append a bucket identifier to hot-key records so their work can be distributed across multiple partitions. For a join, corresponding records on the other side must be replicated or otherwise mapped to matching salt buckets without changing the intended result. For aggregation, compute partial results by the salted key and combine them using the original key. Sum and count combine directly; an average requires combining partial sums and counts rather than averaging partial averages. Salting adds processing, data movement, and implementation complexity, so apply it selectively.

Other remedies include filtering unnecessary records before a shuffle, projecting only required columns, using partial aggregation where supported, isolating hot keys, and choosing a better distribution key. Repartitioning can help when many distinct keys are unevenly assigned, but it adds shuffle overhead and cannot by itself divide one hot key under unchanged hash partitioning. Larger workers may prevent a memory failure without fixing the underlying imbalance, while adding workers often leaves the overloaded partition as the bottleneck.

For ongoing pipeline operations, monitor both end-to-end latency and the spread of task or partition metrics. Streaming skew can concentrate writes on one shard or keyed state on one processing task, causing throttling or growing lag. Changing partition keys can improve distribution but may change ordering guarantees or require downstream reaggregation. Validate every mitigation against output correctness, peak memory, shuffle volume, runtime, and cost using representative data, including known hot keys.

## Learn Data Skew in lessons

1 short lesson on Data Skew in 1 certification course.

- DEA-C01: Data Skew: partition imbalance (Premium), https://app.blowstack.com/lessons/data-skew-partition-imbalance-dea-c01
