Data Compression

Data compression reduces the number of bytes needed to represent information. Lossless compression preserves the original data exactly and is the normal choice for database records, logs, financial transactions, and analytical datasets. Lossy compression discards some information and is appropriate only when the application tolerates that loss, such as certain image, audio, or video workloads. Compression effectiveness depends on the data: repetitive text often compresses well, while encrypted data and already compressed media usually provide little additional reduction.

In cloud pipelines, compression can happen before upload, during transformation, or inside a storage engine. Compressing source files before transfer reduces the bytes crossing the network, which can shorten uploads when bandwidth is the bottleneck. However, it does not automatically reduce every bill. Internet data transfer into Amazon S3 is generally free, so compression primarily saves upload time and stored bytes in that case; source-provider egress charges, connectivity charges, and compression compute costs must also be considered. Compare the total cost of transferring the compressed dataset within the required deadline, rather than assuming that the smallest file always produces the cheapest migration.

File format and compression are separate choices. CSV is a text-based row format, while Apache Parquet is a typed columnar format that supports encodings and compression codecs. Converting CSV to Parquet requires parsing records and assigning appropriate types; simply renaming a file does not transform it. For example, a pipeline can read sales CSV files, validate decimal amounts and timestamps, and write Parquet using Snappy or Zstandard where supported. Columnar layout lets an analytical engine read selected columns, while compression reduces the bytes needed for those columns. Format conversion can change textual representations, so preserve original files when byte-for-byte source fidelity is required.

Choose codecs using representative benchmarks. Snappy commonly favors compression and decompression speed, while Zstandard can provide stronger compression with configurable processing trade-offs. Gzip is widely compatible, but a conventional gzip-compressed CSV file is generally not splittable across workers within that file. A single very large gzip file can therefore limit parallel processing. Parquet with internal compression retains row-group-based parallel read opportunities; wrapping an entire Parquet file in an external gzip stream is generally unsuitable for analytical access.

Compression complements partitioning, indexing, and file sizing rather than replacing them. Partitioning by a frequently filtered date can eliminate irrelevant files, indexes or column statistics can help skip irrelevant data, and compression reduces the remaining bytes read. Excessive partitioning can create many small files. Compacting small records into appropriately sized files reduces object request overhead and improves processing efficiency, but compaction and compression are different operations. Combining records into fewer S3 objects can reduce PUT request counts; compressing each record separately does not. Multipart upload divides one object into upload parts and is not a way to combine many records into fewer objects or inherently lower request costs.

Balance batch size against ingestion latency, retry cost, and available parallelism. An hourly batch may be economical for reporting but unsuitable for alerts that require data within seconds. Measure stored size, upload duration, decompression CPU, query scan volume, and end-to-end latency. Confirm that every producer and consumer supports the selected format and codec, and compress before encryption when the application controls both operations. Compression is not encryption and provides no confidentiality by itself.