Cloud concept · Updated

Data Profiling - Data profiling is the systematic examination of a dataset’s structure, values, distributions, and relationships to understand its characteristics and identify potential quality problems.

In short

Data profiling establishes what data actually contains before engineers transform, integrate, or use it for analytics, machine learning, or foundation model applications. Its findings guide cleansing decisions, validation rules, and monitoring thresholds, reducing the risk that inconsistent or unsuitable data reaches downstream consumers.

Cloud Concepts Hub →

6 key aspects of Data Profiling

Effective data profiling combines statistical discovery, consistency analysis, representative coverage, and repeatable measurements to inform consumer-specific validation and preparation.

01
Structural and statistical discovery

Inspect observed types, missingness, cardinality, ranges, and distributions rather than relying only on declared schemas. Use these findings to select transformations and identify fields requiring investigation.

02
Consistency and relationship analysis

Compare formats, units, keys, and business relationships within and across datasets. For example, detect mixed currencies or orphaned foreign keys before joining sources.

03
Evidence-based validation rules

Translate profile findings into explicit completeness, uniqueness, validity, and consistency requirements. Review inferred thresholds with data owners instead of treating historical patterns as authoritative.

04
Representative coverage

Choose full scans or documented sampling strategies according to dataset size and risk. Profile important partitions and demographic or operational segments separately so aggregate statistics do not conceal localized defects.

05
Repeatable monitoring and quality gates

Persist comparable metrics across pipeline runs and distinguish expected variation from problematic changes. Configure explicit failure, quarantine, and warning actions rather than assuming a report will prevent bad data from advancing.

06
Consumer-specific suitability

Interpret quality in the context of the downstream workload, including analytical joins, ML features, or foundation model documents. Protect sensitive profile outputs and supplement statistical checks with semantic and policy validation.

Cloud services for Data Profiling

Use DataBrew or Data Wrangler to explore datasets and identify preparation requirements, or perform analyses within SageMaker Unified Studio. Convert the findings into reviewed validation rules and repeatable transformations with AWS Glue and AWS Glue Data Quality. Add Lambda checks for bounded, application-specific requirements, such as mandatory document metadata. Publish operational quality metrics to CloudWatch for trend monitoring and alarms. The pipeline must explicitly enforce decisions about rejection, quarantine, or continued processing; profile reports and monitoring dashboards alone do not prevent unsuitable data from reaching downstream systems.

AWS Glue DataBrew

DataBrew profile jobs generate statistics that help identify missing values, unusual distributions, and other quality concerns. Engineers can use the findings to design recipe-based transformations and run repeatable preparation jobs.

AWS GlueAWS Glue

AWS Glue Data Quality provides rule recommendations and rule evaluation for supported data workflows. Glue ETL jobs can combine profiling calculations, transformations, and quality checks; discovering schemas with a crawler alone does not validate data quality.

Amazon SageMaker Data Wrangler

Data Wrangler supports visual data exploration and preparation, including data quality and insights reports. Engineers can inspect distributions, missing values, and potential ML preparation issues before defining transformations.

AWS LambdaAWS Lambda

Lambda can implement bounded custom checks, such as validating document metadata, parsing a small input object, or inspecting a processing manifest. Use distributed processing rather than a single function for dataset-wide scans that exceed function runtime or resource limits.

Amazon CloudWatchAmazon CloudWatch

CloudWatch tracks quality measurements published by processing jobs or custom functions, such as invalid-record counts and missing-value rates. Alarms can notify operators or initiate configured responses, but CloudWatch does not independently profile dataset contents.

Data Profiling exam questions

Real practice exam questions on Data Profiling, with the correct answer and its explanation. Answer them with every option in the exam simulator.

MLA-C02 (beta)

A retailer receives a daily training dataset containing 40 million product records in Amazon S3. Before model training starts, at least 99.5% of records must have a non-null product_id, and every discount_rate value must be between 0 and 1. Invalid values are rare and might occur anywhere in the dataset. Analysts need to maintain the checks through a visual interface without writing validation code. The workflow must assess the delivered dataset without correcting or removing records and block training when either requirement is violated. Which solution best meets these requirements?

Show answerCORRECT ANSWER

Define an AWS Glue DataBrew ruleset requiring at least 99.5% completeness for product_id and all discount_rate values to be within the permitted range. Evaluate it with a profile job configured for the full dataset, and make the workflow start training only when both validation results pass.

DataBrew rulesets provide configurable data quality checks that profile jobs can evaluate without custom validation code. Full-dataset profiling addresses rare violations, and explicitly checking the rule outcomes provides the required training gate without modifying the delivered records.

AIP-C01

A company is preparing 80 million archived chat conversations in Amazon S3 for foundation model fine-tuning. An existing AWS Glue Spark job can read the entire archive. Conversations are partitioned by month and language, and records within each partition are ordered by conversation category. Some categories occur infrequently but contain unusually long conversations.

Before choosing filtering or truncation rules, the developer must determine the exact proportion of conversations that exceed the training pipeline's token-length limit for each language and conversation category. The analysis must also show the median, upper percentiles, and maximum token length for each group. Source conversations must remain unchanged, and statistics must describe the conversations before any content is removed.

Which approach meets these requirements?

Show answerCORRECT ANSWER

Use the Glue Spark job to read every partition, render each conversation using the training pipeline's input format, and count tokens with its tokenizer. Aggregate token-length quantiles, maximums, record counts, and counts above the limit by language and conversation category. Write the resulting statistics to a separate S3 location.

This profiles the complete, unmodified population using the relevant token-length measure. Grouped statistics expose rare categories and language-specific differences, while the counts above the limit and total counts provide exact exceedance proportions. Quantiles can be calculated exactly or with a documented approximation appropriate to the reporting requirements; the exceedance counts remain exact.

DEA-C01

A retailer receives a daily product catalog from a new supplier in Amazon S3. The files have the expected columns and data types, but analysts report unexpected changes in price ranges and product categories. The supplier has not provided reliable quality thresholds. Before choosing cleansing operations, a data engineer must establish the observed characteristics of each delivery, including missing-value percentages, distinct-value counts, and numeric and categorical distributions. The engineer must retain reports for comparison across deliveries, leave the source records unchanged, and avoid writing custom code.

Which approach meets these requirements?

Show answerCORRECT ANSWER

Configure AWS Glue DataBrew profile jobs for each unmodified delivery, using the full dataset. Store the profile outputs in Amazon S3 and compare the column statistics and distributions across deliveries.

DataBrew profile jobs generate dataset and column statistics without requiring transformation code or modifying the source. Profiling each delivery establishes its observed characteristics, and retained reports support comparisons that guide later cleansing decisions and quality thresholds.

Practice Data Profiling in a full exam

Questions on Data Profiling in each practice exam, with every answer option and an explanation for each:

Start a free practice exam

What is Data Profiling

3 min read

Data profiling answers practical questions about a dataset: Which columns contain missing values? Are identifiers unique? Do dates follow a consistent format? Are numeric values plausible? Engineers examine metadata and actual records to calculate statistics such as row counts, null percentages, distinct counts, minimums, maximums, quantiles, and value frequencies. These measurements establish a baseline for understanding data rather than assuming that its documented schema accurately describes its contents.

Profiling can examine individual columns, relationships between columns, and consistency across datasets. For example, a customer feed might contain country codes in several formats, customer identifiers with leading zeros, and timestamps that mix time zones. Cross-column analysis might reveal orders whose shipment dates precede their creation dates. Cross-dataset analysis can identify orders that reference customers absent from the customer table. Such findings inform transformation decisions, including preserving identifiers as strings, normalizing codes, and defining explicit timestamp conversion rules.

In a cloud pipeline, profiling commonly runs after ingestion and before transformation, with additional checks after transformation. Raw objects can remain unchanged in storage while scheduled or event-driven jobs produce profile reports and metrics. Engineers use those results to choose cleansing operations and create repeatable preparation workflows. AWS Glue DataBrew provides profile jobs and recipe-based preparation, while environments such as Amazon SageMaker Unified Studio support inspecting and preparing data through their available analytical tools. A schema crawler alone does not replace profiling: discovering column names and inferred types does not establish completeness, consistency, or business validity.

Profiling and validation are related but distinct. Profiling discovers observed characteristics; validation evaluates data against explicit expectations. Finding that a customer identifier is populated in 98 percent of records is profiling. Requiring at least 99.9 percent completeness is validation. AWS Glue Data Quality can recommend rules from observed data characteristics and evaluate configured rules, but engineers must review recommendations against business requirements. A historically common value is not necessarily correct, and an unusual value is not necessarily an error.

For machine learning, profiles help expose class imbalance, unexpected feature ranges, missing labels, and differences between training and inference inputs. Segment-level analysis can reveal quality problems hidden by aggregate statistics, such as one region having substantially more missing values. These findings support bias investigation, but neither balanced counts nor clean data proves that a model is fair. For foundation model consumption, profile document length, language, duplicate content, extraction failures, and required metadata. Add explicit validation for ingestion formats, permitted content, sensitive information handling, and model-specific size limits before embedding, indexing, or training.

Production workflows turn these observations into quality gates. Distributed processing can calculate dataset-wide metrics, custom Lambda functions can perform bounded checks, and Amazon CloudWatch can track published measurements and trigger alarms. Pipeline logic should explicitly decide whether failures stop processing, quarantine affected records, or generate warnings. Retain profiles, rule versions, and processing identifiers so engineers can investigate changes and reproduce decisions.

The main trade-offs are cost, coverage, and sensitivity. Full scans provide broader coverage but consume more compute and storage I/O; sampling is faster but can miss rare defects or underrepresented groups. Record sample size and selection method, and use full or targeted checks for critical constraints. Protect profile outputs because examples and frequent-value summaries can expose sensitive data. Finally, profiling does not repair data automatically or establish factual accuracy without trusted reference data and domain knowledge.

Learn Data Profiling in lessons

7 short lessons on Data Profiling in 3 certification courses.

and 1 more lesson

Open the lessons