Repartitioning

Repartitioning changes how records are distributed across partitions in a distributed dataset. A partition is a unit of data assigned to a processing task or storage location; it is not necessarily a physical server. Repartitioning can increase or decrease the number of partitions, change the columns used to assign records, or replace an unsuitable distribution strategy. Engineers use it before expensive transformations, when correcting workload imbalance, and when preparing output for downstream consumers.

Data skew occurs when some partitions contain substantially more data or require substantially more processing than others. For example, an events dataset might contain millions of records for one customer but only thousands for each remaining customer. If a join or aggregation partitions records by customer identifier, the busiest customer can dominate one task. Common symptoms include a few long-running tasks, large differences between median and maximum task duration, excessive disk spilling, and memory failures concentrated in particular tasks. Compare partition sizes, shuffle volumes, key frequencies, and execution plans before changing the pipeline.

In distributed processing engines, repartitioning commonly requires a shuffle: workers exchange records over the network according to a new assignment rule. Hash partitioning places records with the same partitioning key together, which supports joins and grouped aggregations. Range partitioning assigns records according to key ranges, while round-robin distribution can spread records more evenly when key colocation is unnecessary. The best choice depends on the next operation, available execution capacity, record sizes, and the distribution of keys.

In a Spark-based cloud job, repartitioning to more partitions can help when many distinct keys have been packed into too few tasks. However, repartitioning by the same customer identifier does not split a single hot customer across multiple hash partitions. Increasing worker count alone also does not resolve that bottleneck. Repartitioning without a key may balance an intermediate stage, but a later customer-based aggregation can introduce the same skew again.

For a heavily skewed aggregation, an engineer can add a salt value that divides the hot customer's records into several subgroups, calculate partial totals by customer and salt, and then combine those totals by customer. The final calculation must preserve the original semantics: an average requires combining sums and counts rather than averaging partial averages. For a skewed join, salting may require replicating matching rows from the smaller input across salt values. Incorrect replication or join conditions can duplicate or omit results. Broadcasting a sufficiently small join input or using supported adaptive skew-join optimization may avoid manual salting.

Repartitioning has costs. Shuffles consume network bandwidth, CPU, memory, and temporary disk space. Too many partitions increase scheduling overhead and can produce small output files; too few limit parallelism and increase per-task memory pressure. In Spark, coalesce typically reduces partition count without a full shuffle, making it cheaper but less suitable for correcting an uneven distribution. Processing partitions also differ from table directory partitions in object storage: changing one does not automatically optimize the other.

Validate both performance and correctness after a change. Compare task-duration distributions, spilling, shuffle bytes, runtime, and cost, then verify row counts, business totals, null handling, and join cardinality. Repartitioning supports reliable data-quality processing by reducing failures and incomplete runs, but it does not itself correct invalid values, remove duplicates, or guarantee balanced workloads.