Cloud concept · Updated

Apache Parquet - Apache Parquet is an open, column-oriented binary file format designed for efficient storage and analytical processing of structured and nested data.

In short

Parquet groups values by column, allowing compatible readers to load selected fields and skip irrelevant data rather than scan every record in full. Its typed schema, encodings, and compression make it useful for cloud data lakes and machine learning preparation, especially when workloads repeatedly analyze subsets of large datasets.

Cloud Concepts Hub →

6 key aspects of Apache Parquet

Parquet combines selective column reads, typed schemas, compression, and metadata-based skipping, with performance determined partly by dataset layout and reader capabilities.

01
Column-oriented access

Readers can select required columns instead of loading every field. This is especially useful for feature engineering and analytical queries over wide datasets.

02
Typed and nested representation

Files carry schema information and support nested structures such as lists and structs. Validate logical types and cross-tool compatibility, particularly for timestamps and decimals.

03
Encoding and compression

Column-specific encodings and page compression can reduce storage and transfer volume. Select supported codecs and benchmark their CPU, size, and read-performance trade-offs.

04
Metadata-assisted data skipping

Compatible readers can use statistics to avoid reading irrelevant row groups or pages. Sorting or clustering related values can make this skipping more effective.

05
Dataset layout and parallelism

Partitioning, file sizes, and row-group sizes influence scan efficiency and processing parallelism. Compact small files and avoid creating excessive partitions.

06
File format rather than table management

Parquet does not independently provide dataset-wide schema enforcement, transactional updates, or record indexing. Use appropriate processing engines and, when required, a transactional table format.

Cloud services for Apache Parquet

A typical workflow stores raw records in Amazon S3, then uses AWS Glue or Amazon EMR to validate, transform, and write partitioned Parquet datasets. The AWS Glue Data Catalog supplies table metadata that Amazon Athena can use for SQL exploration and validation. SageMaker Processing can consume the curated files for feature preparation, with conversion when a training algorithm requires another format. Across these services, efficient access depends on selecting required columns, applying useful filters, maintaining compatible schemas, and controlling partition and file sizes rather than merely changing the file extension.

Amazon S3Amazon S3

Amazon S3 stores Parquet objects as part of a data lake. Organize object prefixes around useful partition keys, but remember that S3 itself does not interpret Parquet columns or execute analytical queries.

AWS GlueAWS Glue

AWS Glue ETL jobs can transform source data into Parquet and compact or repartition datasets. The AWS Glue Data Catalog records table schemas and locations for compatible query engines; catalog metadata does not replace validation of the underlying files.

Amazon AthenaAmazon Athena

Amazon Athena queries Parquet datasets in Amazon S3 using SQL. Selecting only needed columns and using effective partition filters can reduce scanned data and associated query costs.

Amazon EMRAmazon EMR

Amazon EMR runs engines such as Apache Spark that read and write Parquet for distributed preparation and analytics. Engineers can tune partitioning, row groups, codecs, and output file sizes for their workloads.

Amazon SageMakerAmazon SageMaker

SageMaker Processing jobs can use compatible libraries or frameworks to read Parquet and prepare training datasets. Parquet support for training depends on the selected algorithm or container, so convert the output when its input contract requires another format.

Apache Parquet exam questions

Real practice exam questions on Apache Parquet, with the correct answer and its explanation. Answer them with every option in the exam simulator.

MLA-C02 (beta)

A company retains raw customer events as JSON in Amazon S3. An AWS Glue job writes a curated Apache Parquet dataset that downstream Spark jobs use to prepare ML features. Historical files store customer_id as a 64-bit integer. A new source sends customer identifiers such as "00127" and "A912", so the latest files store customer_id as a string. Downstream jobs now fail when reading historical and recent files together because of incompatible schemas.

The company must retain all historical records, preserve customer identifiers exactly as represented in the raw events, and provide one consistently typed dataset for future feature preparation. All original raw events are still available.

Which change should an ML engineer make?

Show answerCORRECT ANSWER

Reprocess all retained raw events with an explicit schema that treats customer_id as a string, write a replacement Parquet dataset, and enforce that schema for future writes.

Rebuilding from the raw events preserves identifier text, including leading zeros and alphabetic characters. Writing every file with a consistent string type removes the conflicting Parquet schemas. Enforcing the same schema on future writes prevents the conflict from recurring.

SAA-C03

An insurance company receives daily UTF-8 CSV exports in Amazon S3. Each file has a header row. Policy identifiers contain only digits but can begin with zeros that must be preserved. Customer address fields are enclosed in double quotes and can contain commas and embedded line breaks. The company needs a serverless batch transformation that creates a separate Apache Parquet dataset while preserving every identifier and keeping each address in a single field. Which solution meets these requirements?

Show answerCORRECT ANSWER

Run an AWS Glue Spark job with a CSV reader configured for headers, double-quote handling, and multiline records. Provide an explicit schema that reads policy identifiers as strings, and write the results as Parquet to a separate S3 prefix.

A CSV-aware reader preserves commas inside quoted fields, and multiline support preserves embedded line breaks within an address rather than treating them as new records. Reading identifiers as strings from the start preserves leading zeros. AWS Glue provides serverless batch processing and can write a separate Parquet dataset.

Practice Apache Parquet in a full exam

Questions on Apache Parquet in each practice exam, with every answer option and an explanation for each:

Start a free practice exam

What is Apache Parquet

3 min read

Apache Parquet is a file format, not a database, query engine, or storage service. It defines how records, schemas, and metadata are represented inside files. Engineers commonly store these files in cloud object storage and process them with distributed engines, SQL query services, or libraries such as Apache Arrow. The same dataset can therefore support multiple processing tools without requiring a proprietary storage format.

A Parquet file divides records into row groups. Within each row group, values for each column are stored in column chunks, which contain pages. This organization supports column projection: a feature engineering job that needs five fields from a table containing two hundred fields can read the relevant column data rather than all fields. Repeated values and similar data types also work well with encodings and compression codecs such as Snappy and Zstandard. Actual savings depend on the data, writer settings, and reader implementation.

Readers can use metadata, including available minimum and maximum statistics, to skip row groups that cannot satisfy a filter. For example, a job filtering transactions by timestamp may avoid reading row groups whose timestamp ranges fall entirely outside the requested interval. This benefit depends on reader support and data organization; randomly distributed timestamps may make every row group overlap the filter. Predicate pushdown does not mean that every filter automatically eliminates physical reads.

In a cloud ML pipeline, incoming events might first arrive as JSON because producers need flexible, self-describing records. A preparation job can validate types, normalize timestamps, handle missing values, and write curated Parquet files partitioned by event date. Later jobs can read only the required dates and feature columns. Directory partition pruning and Parquet row-group pruning are separate optimizations: the former selects files using dataset layout, while the latter uses metadata inside files.

Choose formats according to access patterns. CSV is convenient for simple interchange and systems requiring flat text, but it lacks a standardized embedded typed schema. JSON suits nested event payloads and application interchange but often requires more parsing and storage than a well-written Parquet equivalent. ORC is another columnar analytical format; choose between ORC and Parquet based on engine support, interoperability, and measured performance rather than assuming one always wins. Parquet is particularly useful for repeated scans, aggregations, and feature preparation, but it is not designed for low-latency individual-record updates.

File layout matters operationally. Thousands of tiny files increase listing, request, scheduling, and metadata overhead, while excessively large files or row groups can reduce useful parallelism or increase memory pressure. Compact small outputs and tune file sizes, row-group sizes, partition keys, and compression against representative workloads. Avoid partitioning by identifiers with extremely high cardinality unless the access pattern justifies the resulting layout.

Parquet stores schema information, but it does not automatically enforce a consistent schema across every file in a dataset. Validate changes to field types, decimal precision, and timestamp interpretation across writers and readers. Parquet alone also does not provide table-level transactions or coordinated updates; table formats can add those capabilities. Finally, confirm the consuming ML framework or algorithm accepts Parquet. A preparation step may still need to convert the curated dataset into a training-specific format.

Learn Apache Parquet in lessons

4 short lessons on Apache Parquet in 2 certification courses.

Open the lessons