Cloud concept · Updated

Data Lakes - A data lake is a shared repository that stores large volumes of structured, semi-structured, and unstructured data for analytics and other processing.

In short

Data lakes retain data in its original form and in prepared formats so different teams can reuse it for reporting, exploration, and machine learning. In the cloud, object storage commonly provides the storage foundation, while separate services handle ingestion, cataloging, governance, and computation.

Cloud Concepts Hub →

6 key aspects of Data Lakes

Data lakes combine flexible data retention with independent processing, metadata, governance, and deliberate preparation for reliable analytics.

01
Storage for diverse data

A lake can retain relational exports, event records, documents, and media in one logical environment. Keeping original inputs allows teams to reprocess data when requirements or transformation rules change.

02
Object storage foundation

Cloud lakes commonly use object storage for durable, scalable data retention. Storage remains available independently of the compute engines that process or query it.

03
Flexible schema application

Schemas can be applied when data is read, allowing multiple interpretations of retained inputs. Production datasets still need validation, documented semantics, and controlled schema evolution.

04
Cataloging and governance

Metadata makes datasets discoverable, while ownership, access controls, and retention rules make them manageable. These controls distinguish a usable analytical repository from an unmanaged collection of files.

05
Raw and curated datasets

Separating original inputs from validated and analytics-ready outputs helps users select appropriate data. It also supports troubleshooting and repeatable transformations.

06
Independent processing and cost optimization

Different query, batch-processing, and machine learning engines can use the same stored datasets. File formats, partitioning, storage tiers, and scan volume affect both performance and total cost.

Cloud services for Data Lakes

Amazon S3 holds the lake's raw and curated objects. AWS Glue jobs can validate and transform incoming data, while the Glue Data Catalog records table definitions and dataset locations. Amazon Athena uses that metadata to query supported data in S3 with SQL. AWS Lake Formation helps administer permissions for supported access paths. Together, these services separate storage, preparation, discovery, querying, and governance rather than treating a storage bucket as a complete analytics system. Engineers still need to design ingestion pipelines, data quality checks, access policies, and efficient file layouts.

Amazon S3Amazon S3

Amazon S3 provides the object storage foundation for a cloud data lake, holding raw and curated datasets in buckets. It stores the data but does not itself replace a catalog, transformation engine, or SQL query engine.

AWS GlueAWS Glue

AWS Glue provides data integration and transformation capabilities for preparing lake datasets. Its Data Catalog stores metadata that compatible analytics engines use to locate and interpret data.

Amazon AthenaAmazon Athena

Amazon Athena can query supported datasets in Amazon S3 using SQL without provisioning query servers. Columnar formats, compression, and partition-aware queries can reduce the data scanned and improve efficiency.

AWS Lake FormationAWS Lake Formation

AWS Lake Formation helps govern data lakes through centralized permissions for cataloged data accessed by supported services. Its permissions must be configured alongside appropriate IAM and S3 access controls.

Data Lakes exam questions

Real practice exam questions on Data Lakes, with the correct answer and its explanation. Answer them with every option in the exam simulator.

SAA-C03

A manufacturer collects JSON equipment events, relational maintenance exports, and binary vibration recordings from its factories. Event producers frequently add optional fields. Analysts need standardized datasets for SQL reporting, while data scientists need the original recordings and event fields to develop new failure-detection models. The company must be able to reprocess the exact original inputs for 2 years because factories delete their local copies after delivery. Storage must scale independently of analytical compute, and new optional fields must not interrupt ingestion.

Which architecture best meets these requirements?

Show answerCORRECT ANSWER

Store original inputs in an Amazon S3 raw area for 2 years. Use AWS Glue to catalog supported datasets and transform reporting data into a separate curated area. Allow analytical engines and machine learning jobs to access the appropriate datasets without modifying the originals.

This data lake architecture preserves diverse original inputs for future processing while providing separate, standardized datasets for reporting. New optional fields can be retained without blocking raw ingestion. S3 storage scales independently of the compute used for transformation, SQL analysis, and machine learning.

SCS-C03

A company uses AWS Organizations to manage 60 accounts with workloads in three AWS Regions. The security team must collect CloudTrail management events and VPC Flow Logs from every account in all three Regions. Copies of all collected records must be consolidated into company-controlled object storage in one designated analysis Region, although Regional copies may remain.

The consolidated datasets must use a common security schema and a columnar format. A third-party security platform will analyze the datasets through an existing supported integration. The company wants managed collection, normalization, cataloging, and cross-Region consolidation without maintaining custom pipelines.

Which solution meets these requirements?

Show answerCORRECT ANSWER

Designate an Amazon Security Lake delegated administrator. Enable the required sources for all organization accounts in each workload Region, and configure the analysis Region as a rollup Region with the other Regions as contributing Regions.

Security Lake provides managed collection of the supported sources, normalizes them to OCSF, stores them as Parquet objects, and maintains catalog metadata. Enabling collection in every workload Region establishes coverage, while the rollup configuration consolidates copies into the analysis Region for independent analytics tools to consume.

Practice Data Lakes in a full exam

Questions on Data Lakes in each practice exam, with every answer option and an explanation for each:

Start a free practice exam

What is Data Lakes

2 min read

A data lake stores diverse datasets for analysis without requiring every dataset to fit a single predefined relational model before ingestion. It can contain structured database exports, semi-structured JSON events, and unstructured documents, images, or audio. Organizations commonly retain original inputs alongside cleaned and transformed datasets so they can investigate historical events or apply new processing logic later.

Cloud object storage is a common foundation because it supports large quantities of data, durable storage, and access by multiple processing systems. Objects reside in buckets or containers and are identified by keys rather than accessed as blocks on a server disk. Storage capacity can grow independently of the compute resources used to analyze it. For AWS Cloud Practitioner questions, a repository for large volumes of raw data used in analytics is a typical object storage use case, with Amazon S3 providing the storage foundation.

Consider a retailer collecting website click events, order exports, and product photographs. An ingestion pipeline writes these inputs into a raw area of the lake. Processing jobs validate records, remove duplicates, and create curated datasets. Analysts query prepared order and click data to understand purchasing behavior, while machine learning jobs can reuse the photographs and historical records. Separate buckets or key prefixes often distinguish raw, validated, and curated data.

Data lakes frequently use schema-on-read: a processing engine applies an interpretation of the data when it reads it. This differs from requiring all incoming records to conform to a warehouse schema before loading. However, schema-on-read does not mean schemas are unnecessary. Reliable analysis still requires documented field meanings, compatible data types, validation rules, and management of schema changes. Curated lake datasets often have strongly enforced structures.

A useful lake also needs a metadata catalog that records dataset locations, schemas, ownership, and other descriptive information. Access controls should restrict sensitive datasets to authorized users and workloads. Encryption, access logging, retention policies, and data classification support security and governance. Copying every available file into a bucket without ownership or quality controls can create a data swamp: a collection that is difficult to discover, trust, or use.

Low storage cost does not guarantee low overall cost or fast queries. Repeatedly scanning large raw files can increase processing time and charges. Compression, columnar formats such as Parquet, sensible partitioning, and avoiding excessive numbers of tiny files can improve efficiency. Lifecycle policies can reduce the cost of older data, but archival tiers may require restoration before interactive analysis is possible.

A data lake is an architecture, not a database engine, and object storage alone does not provide SQL queries or row-level transactions. Query and processing engines supply those capabilities; specialized table formats can add transaction support when used with compatible engines. A lake can complement a data warehouse rather than replace it, and it is generally not the direct substitute for a transactional database serving frequent, low-latency application updates.

Learn Data Lakes in lessons

3 short lessons on Data Lakes in 3 certification courses.

Open the lessons