Data Lakes

A data lake stores diverse datasets for analysis without requiring every dataset to fit a single predefined relational model before ingestion. It can contain structured database exports, semi-structured JSON events, and unstructured documents, images, or audio. Organizations commonly retain original inputs alongside cleaned and transformed datasets so they can investigate historical events or apply new processing logic later.

Cloud object storage is a common foundation because it supports large quantities of data, durable storage, and access by multiple processing systems. Objects reside in buckets or containers and are identified by keys rather than accessed as blocks on a server disk. Storage capacity can grow independently of the compute resources used to analyze it. For AWS Cloud Practitioner questions, a repository for large volumes of raw data used in analytics is a typical object storage use case, with Amazon S3 providing the storage foundation.

Consider a retailer collecting website click events, order exports, and product photographs. An ingestion pipeline writes these inputs into a raw area of the lake. Processing jobs validate records, remove duplicates, and create curated datasets. Analysts query prepared order and click data to understand purchasing behavior, while machine learning jobs can reuse the photographs and historical records. Separate buckets or key prefixes often distinguish raw, validated, and curated data.

Data lakes frequently use schema-on-read: a processing engine applies an interpretation of the data when it reads it. This differs from requiring all incoming records to conform to a warehouse schema before loading. However, schema-on-read does not mean schemas are unnecessary. Reliable analysis still requires documented field meanings, compatible data types, validation rules, and management of schema changes. Curated lake datasets often have strongly enforced structures.

A useful lake also needs a metadata catalog that records dataset locations, schemas, ownership, and other descriptive information. Access controls should restrict sensitive datasets to authorized users and workloads. Encryption, access logging, retention policies, and data classification support security and governance. Copying every available file into a bucket without ownership or quality controls can create a data swamp: a collection that is difficult to discover, trust, or use.

Low storage cost does not guarantee low overall cost or fast queries. Repeatedly scanning large raw files can increase processing time and charges. Compression, columnar formats such as Parquet, sensible partitioning, and avoiding excessive numbers of tiny files can improve efficiency. Lifecycle policies can reduce the cost of older data, but archival tiers may require restoration before interactive analysis is possible.

A data lake is an architecture, not a database engine, and object storage alone does not provide SQL queries or row-level transactions. Query and processing engines supply those capabilities; specialized table formats can add transaction support when used with compatible engines. A lake can complement a data warehouse rather than replace it, and it is generally not the direct substitute for a transactional database serving frequent, low-latency application updates.