Cloud concept · Updated

Data Compression - Data compression represents information using fewer bytes to reduce storage consumption and data movement.

In short

Compression reduces the physical size of data by encoding it more efficiently, usually without losing information in engineering and analytics workloads. In cloud architectures, it can lower storage and query costs and improve transfer throughput, but it introduces processing overhead and must match the workload's access patterns.

Cloud Concepts Hub →

6 key aspects of Data Compression

Effective compression preserves required information while balancing byte reduction, processing overhead, compatibility, parallel access, and total workload cost.

01
Lossless preservation

Use lossless compression when records must be reconstructed exactly. Treat lossy compression as an explicit application-level decision, not a routine optimization for analytical records.

02
Codec and compute trade-offs

Compression ratio, encoding speed, decompression speed, and memory consumption vary by codec and dataset. Benchmark realistic data and include compute costs rather than optimizing only for the smallest output.

03
Format-aware optimization

Columnar formats such as Parquet combine typed storage and column encodings with optional compression. Converting CSV to compressed Parquet can reduce both stored bytes and the amount of data analytical queries need to read.

04
Parallel access and splittability

Compression boundaries affect whether multiple workers can process a file concurrently. Prefer layouts with independently readable units for distributed analytics, and avoid a single large, nonsplittable compressed text file when parallelism matters.

05
Complementary data layout

Combine compression with selective partitioning, useful indexes or statistics, and suitable file sizes. These techniques reduce different sources of waste and should be evaluated together.

06
End-to-end cost evaluation

Account for storage, requests, network pricing, transformation compute, and latency. Batching records into fewer objects reduces request overhead, while compression reduces bytes; neither guarantees the lowest total cost independently.

Cloud services for Data Compression

AWS Glue can transform raw files into typed, compressed Parquet stored in Amazon S3. Amazon Data Firehose provides a streaming ingestion path that buffers records and can produce supported compressed or columnar output. Amazon Athena queries these S3 datasets directly, benefiting from compression, column selection, and partition pruning. Amazon Redshift applies its own column encodings when data is loaded into warehouse tables. Across these paths, choose compatible codecs and practical file sizes, and measure ingestion latency, transformation compute, storage consumption, and query costs together. S3 preserves the object bytes supplied by the producer; it does not perform the compression itself.

AWS GlueAWS Glue

AWS Glue ETL jobs can parse CSV data and write compressed Parquet, with explicit schema and codec choices. Jobs can also reorganize data into partitions and compact small files to improve downstream query efficiency.

Amazon S3Amazon S3

Amazon S3 stores compressed objects but does not automatically compress uploaded application data. Compress before upload or write compressed output from a processing pipeline, and evaluate object sizes, request counts, and storage-class billing rules.

Amazon AthenaAmazon Athena

Amazon Athena can query supported compressed files and columnar formats in S3. For queries billed by data scanned, compressed Parquet combined with column selection and partition pruning can reduce scan charges compared with scanning uncompressed CSV.

Amazon RedshiftAmazon Redshift

Amazon Redshift uses column compression encodings to reduce storage and I/O, with automatic encoding management available through ENCODE AUTO. Table compression is separate from the compression of source files used during loading.

Data Compression exam questions

Real practice exam questions on Data Compression, with the correct answer and its explanation. Answer them with every option in the exam simulator.

DEA-C01

A company uses Apache Spark on Amazon EMR to convert nightly CSV exports in Amazon S3 into Parquet. Each export arrives as one 600 GB gzip-compressed CSV object. Monitoring shows that the input stage uses one task to read and decompress the object while most executors remain idle. This stage accounts for most of the job's runtime.

The source exporter can change how it packages future exports. The company must retain gzip compression for the source objects and preserve every CSV record. Which change will allow multiple Spark tasks to read and decompress each export concurrently?

Show answerCORRECT ANSWER

Divide each export at CSV record boundaries and write the resulting portions as multiple independently gzip-compressed S3 objects.

Gzip-compressed CSV objects are not splittable by Spark's standard file reader. Creating multiple independently compressed objects provides separate input units that Spark tasks can read and decompress concurrently while preserving all records.

SAA-C03

An engineering company transfers a new 1,000-GB simulation dataset from an on-premises compute cluster to Amazon S3 each week. The company's network provider charges $0.10 per GB uploaded. Each dataset must be recoverable byte for byte, and downstream applications support both compression settings that the company has tested.

Benchmarks show the following additional processing costs and output sizes for each dataset:
• Fast lossless setting: $15 in processing costs; 200 GB of output.
• Maximum lossless setting: $80 in processing costs; 100 GB of output.

Uploading the original dataset requires no additional preprocessing. All approaches can finish within the transfer window. The company wants to minimize the combined preprocessing and network transfer costs per dataset.

Which solution should a solutions architect recommend?

Show answerCORRECT ANSWER

Apply the fast lossless setting on the source cluster, and upload the resulting files to S3.

The total cost is $15 for processing plus $20 for transferring 200 GB, or $35 per dataset. This is the lowest combined cost. Lossless compression preserves every original byte, and the applications can read the resulting files.

Practice Data Compression in a full exam

Questions on Data Compression in each practice exam, with every answer option and an explanation for each:

Start a free practice exam

What is Data Compression

3 min read

Data compression reduces the number of bytes needed to represent information. Lossless compression preserves the original data exactly and is the normal choice for database records, logs, financial transactions, and analytical datasets. Lossy compression discards some information and is appropriate only when the application tolerates that loss, such as certain image, audio, or video workloads. Compression effectiveness depends on the data: repetitive text often compresses well, while encrypted data and already compressed media usually provide little additional reduction.

In cloud pipelines, compression can happen before upload, during transformation, or inside a storage engine. Compressing source files before transfer reduces the bytes crossing the network, which can shorten uploads when bandwidth is the bottleneck. However, it does not automatically reduce every bill. Internet data transfer into Amazon S3 is generally free, so compression primarily saves upload time and stored bytes in that case; source-provider egress charges, connectivity charges, and compression compute costs must also be considered. Compare the total cost of transferring the compressed dataset within the required deadline, rather than assuming that the smallest file always produces the cheapest migration.

File format and compression are separate choices. CSV is a text-based row format, while Apache Parquet is a typed columnar format that supports encodings and compression codecs. Converting CSV to Parquet requires parsing records and assigning appropriate types; simply renaming a file does not transform it. For example, a pipeline can read sales CSV files, validate decimal amounts and timestamps, and write Parquet using Snappy or Zstandard where supported. Columnar layout lets an analytical engine read selected columns, while compression reduces the bytes needed for those columns. Format conversion can change textual representations, so preserve original files when byte-for-byte source fidelity is required.

Choose codecs using representative benchmarks. Snappy commonly favors compression and decompression speed, while Zstandard can provide stronger compression with configurable processing trade-offs. Gzip is widely compatible, but a conventional gzip-compressed CSV file is generally not splittable across workers within that file. A single very large gzip file can therefore limit parallel processing. Parquet with internal compression retains row-group-based parallel read opportunities; wrapping an entire Parquet file in an external gzip stream is generally unsuitable for analytical access.

Compression complements partitioning, indexing, and file sizing rather than replacing them. Partitioning by a frequently filtered date can eliminate irrelevant files, indexes or column statistics can help skip irrelevant data, and compression reduces the remaining bytes read. Excessive partitioning can create many small files. Compacting small records into appropriately sized files reduces object request overhead and improves processing efficiency, but compaction and compression are different operations. Combining records into fewer S3 objects can reduce PUT request counts; compressing each record separately does not. Multipart upload divides one object into upload parts and is not a way to combine many records into fewer objects or inherently lower request costs.

Balance batch size against ingestion latency, retry cost, and available parallelism. An hourly batch may be economical for reporting but unsuitable for alerts that require data within seconds. Measure stored size, upload duration, decompression CPU, query scan volume, and end-to-end latency. Confirm that every producer and consumer supports the selected format and codec, and compress before encryption when the application controls both operations. Compression is not encryption and provides no confidentiality by itself.

Learn Data Compression in lessons

4 short lessons on Data Compression in 2 certification courses.

Open the lessons