Cloud concept · Updated

Data Cataloging - Data cataloging is the practice of maintaining a searchable inventory of data assets and metadata describing their meaning, ownership, origins, and permitted use.

In short

A data catalog helps engineers discover datasets and determine whether they are appropriate for analytics, model training, evaluation, or retrieval. By documenting provenance, classifications, and usage restrictions, it supports trustworthy AI development, security reviews, and compliance without replacing access controls or data validation.

Cloud Concepts Hub →

6 key aspects of Data Cataloging

Effective data cataloging combines discoverability, ownership, provenance, governance metadata, and ongoing maintenance to make data use more accountable.

01
Searchable asset inventory

Register data locations, schemas, formats, and descriptions so engineers can find existing assets instead of creating duplicate datasets. Use stable asset identifiers even when physical storage locations change.

02
Business context and accountable ownership

Record business definitions, responsible owners, refresh expectations, and known limitations. Route questions and metadata corrections to an identified steward rather than leaving entries without maintenance responsibility.

03
Origins and lineage connections

Document source systems, collection methods, extraction dates, and dataset versions. Link entries to transformation and pipeline records so teams can trace derived datasets back to their origins.

04
Governance and permitted use

Capture sensitivity labels, retention requirements, licensing conditions, and approved purposes such as analytics or model training. Treat these records as inputs to policy enforcement, not as enforcement mechanisms themselves.

05
AI evidence traceability

Connect training, evaluation, and retrieval assets to model documentation and source references. Preserve document and dataset versions so investigations and citations refer to the evidence actually used.

06
Metadata freshness and protection

Refresh technical metadata automatically and review business metadata when ownership or policies change. Restrict catalog visibility where descriptions, schemas, or samples could expose sensitive information.

Cloud services for Data Cataloging

AWS Glue Data Catalog maintains technical metadata used by data processing and query tools. Amazon DataZone adds business-oriented discovery and publishing workflows, while AWS Lake Formation governs access to supported data lake resources. For AI projects, Amazon SageMaker AI adds model documentation and machine learning lineage records that teams can connect to dataset metadata. Together, these capabilities help engineers discover data, understand its origins, request appropriate access, and document how it contributes to a model. Teams must still maintain accurate metadata, retain exact dataset versions, and implement application-level source citations where generated answers require evidence.

AWS GlueAWS Glue

The AWS Glue Data Catalog stores technical metadata such as table definitions, schemas, partitions, and data locations. AWS Glue crawlers can discover supported data sources and populate this metadata, while teams add context that automated discovery cannot determine.

Amazon DataZone

Amazon DataZone provides a business data catalog with asset discovery, business context, and workflows for publishing data and requesting access. Its lineage capabilities can help users understand supported relationships between source data and derived assets.

AWS Lake FormationAWS Lake Formation

AWS Lake Formation uses AWS Glue Data Catalog resources as part of governing data lake access. It complements cataloging with permissions for supported data access paths; catalog descriptions alone do not enforce those permissions.

Amazon SageMakerAmazon SageMaker

Amazon SageMaker Model Cards document model-level information such as intended use, training details, evaluation results, and limitations. Teams can reference cataloged datasets and provenance records in this documentation, while SageMaker lineage tracking can capture relationships among supported machine learning workflow artifacts.

Data Cataloging exam questions

Real practice exam questions on Data Cataloging, with the correct answer and its explanation. Answer them with every option in the exam simulator.

DEA-C01

A company uses Amazon DataZone as its business data catalog. A producer project publishes AWS Glue Data Catalog tables whose Amazon S3 data is governed by AWS Lake Formation. Analysts can discover the published tables, read business descriptions, and submit subscription requests from consumer projects.

Data owners can approve requests, but automatic access provisioning fails. The provisioning logs show that the producer environment's manage-access IAM role cannot grant SELECT permissions to the consumer environment role. The manage-access role has SELECT on the tables without the grant option. Its IAM policy already allows the required Lake Formation API actions, and all other integration permissions are correctly configured.

The company must retain owner approval and automatically provision access for approved subscriptions without maintaining grants for individual consumers. Which change should a data engineer make?

Show answerCORRECT ANSWER

Give the producer environment's manage-access role Lake Formation SELECT permission with the grant option on the published tables.

DataZone uses the environment's manage-access role to provision Lake Formation permissions for supported managed assets after subscription approval. SELECT without the grant option lets the role read data but does not let it delegate SELECT to consumer environment roles. Adding the grant option addresses the identified failure while preserving the catalog's approval and automatic provisioning workflow.

Practice Data Cataloging in a full exam

Questions on Data Cataloging in each practice exam, with every answer option and an explanation for each:

Start a free practice exam

What is Data Cataloging

2 min read

Data cataloging organizes information about data rather than merely collecting the data itself. A catalog entry typically identifies a dataset's location, schema, business meaning, owner, update frequency, and sensitivity classification. It can also record the originating system, collection method, applicable license, retention requirements, and approved uses. The underlying assets may remain in object storage, databases, warehouses, or external systems while the catalog provides a shared discovery and documentation layer.

In cloud environments, teams populate catalogs through automated discovery, database connectors, pipeline integrations, and manual stewardship. Automated discovery can infer column names, types, partitions, and storage locations. However, it usually cannot reliably establish why data was collected, whether consent permits model training, or what a business term means. Engineers should combine discovered technical metadata with reviewed business and governance metadata, and update both when datasets or policies change.

For AI workloads, documenting origins helps teams select appropriate training and evaluation data and investigate problematic outputs. For example, a team preparing support-ticket data for a classifier might register the source application, extraction date, dataset version, personal-information classification, redaction procedure, and permitted purpose. The training pipeline should reference the exact dataset snapshot rather than only a mutable storage path. This makes later investigations more reproducible and helps identify models affected by a faulty or withdrawn source.

Data cataloging, data lineage, and source citation are related but different. A catalog describes assets and helps users find them. Lineage records relationships and transformations, such as a raw table being filtered into a training dataset and then used by a training job. Source citation identifies the evidence supporting a specific statement or generated answer. In a retrieval-augmented generation application, catalog metadata can preserve document ownership, origin, and usage restrictions, while the retrieval pipeline must retain document identifiers, versions, and passage references so answers can cite the actual retrieved evidence. Cataloging alone neither produces citations nor proves that an answer is supported.

Model documentation provides another complementary layer. Amazon SageMaker Model Cards can document a model's intended uses, training details, evaluation results, and limitations. They are model-level records, not replacements for a dataset catalog or a complete lineage system. Linking model documentation to cataloged dataset versions and pipeline records provides a clearer chain from source data to model behavior.

The main trade-off is maintenance effort versus useful coverage. Excessively detailed mandatory fields can discourage registration, while shallow or stale entries can mislead users. Prioritize critical datasets, assign accountable owners, automate updates, and review sensitive classifications. Protect catalog metadata itself because names, descriptions, and sample values can reveal confidential information. Finally, finding a dataset in a catalog does not grant access, certify its quality, establish legal permission, or guarantee regulatory compliance; those outcomes require separate controls and verification.

Learn Data Cataloging in lessons

4 short lessons on Data Cataloging in 2 certification courses.

Open the lessons