Cloud concept · Updated

Data Quality - Data quality is the degree to which data is accurate, complete, consistent, valid, timely, and suitable for its intended use.

In short

Data quality determines whether engineers can rely on data for analytics, application behavior, and AI training or inference. In secure data engineering, quality assessments work alongside privacy protections, access controls, and integrity checks to reduce unreliable outputs, unauthorized changes, and inappropriate use of sensitive information.

Cloud Concepts Hub →

6 key aspects of Data Quality

Reliable data requires measurable, purpose-specific checks backed by continuous monitoring, traceability, secure access, and privacy-aware processing.

01
Purpose-specific acceptance criteria

Define quality expectations around the consuming workload and its risks. A reporting dataset and a real-time decision system may need different freshness limits and error tolerances.

02
Measurable quality dimensions

Measure accuracy, completeness, consistency, validity, uniqueness, and timeliness separately. Passing a schema check does not establish that values are factually correct.

03
Continuous validation and remediation

Run automated checks at ingestion and transformation boundaries, then monitor trends. Assign owners and define when failures should block publication, quarantine records, or trigger investigation.

04
Provenance and protected integrity

Track sources, dataset versions, and transformations while restricting and auditing write access. Use integrity mechanisms to detect unauthorized or accidental changes without confusing unchanged data with accurate data.

05
Privacy-aware processing

Minimize sensitive data and apply appropriate masking, tokenization, or privacy techniques. Reassess quality after these transformations because they can change analytical utility and join behavior.

06
AI dataset suitability

Assess label reliability, representative coverage, and training-evaluation separation. Monitor production input changes and investigate their effect on model performance rather than assuming every change is harmful.

Cloud services for Data Quality

A practical workflow stores source data in Amazon S3 and uses AWS Glue to transform it and evaluate quality rules before publishing curated datasets. Failed records can be isolated for investigation, while versions and validation results support reproducibility. AWS Lake Formation controls supported access to curated lake data, and Amazon Macie helps identify sensitive information requiring additional protection. For deployed AI workloads, Amazon SageMaker AI provides monitoring capabilities that can flag input deviations from established baselines. Together, these services support validation, controlled access, privacy discovery, and ongoing monitoring, but engineers must still define acceptance criteria, investigate exceptions, and verify business correctness.

AWS GlueAWS Glue

AWS Glue Data Quality supports defining and evaluating quality rules for cataloged data and data processed in Glue ETL jobs. Engineers can use rule results to identify completeness, uniqueness, and other quality failures and configure pipeline responses.

Amazon S3Amazon S3

Amazon S3 can store source, quarantined, and curated datasets with separate access policies. Object versioning and checksums support recovery and integrity verification, but additional validation is required to determine whether stored records are correct.

AWS Lake FormationAWS Lake Formation

AWS Lake Formation provides fine-grained permissions for supported access to data lake tables and columns, including row- and cell-level filtering where supported. It helps restrict sensitive data exposure and access to curated datasets, while quality rules remain a separate responsibility.

Amazon MacieAmazon Macie

Amazon Macie discovers sensitive data in Amazon S3, helping engineers identify information that needs restricted access or privacy-preserving transformation. It does not cleanse records, anonymize datasets, or certify their quality.

Data Quality exam questions

Real practice exam questions on Data Quality, with the correct answer and its explanation. Answer them with every option in the exam simulator.

AIB-C01 (beta)

A commercial lender is preparing an AI model to assess loan applications. Historical testing shows excellent accuracy, but a review finds that the training pipeline attaches each applicant's most recently revised financial statement to every past application. Some revisions were published only after the historical lending decision. All statements are accurate, and schema and completeness checks pass. Leadership requires credible evidence of model reliability before deployment. Which approach should the AI business strategist recommend?

Show answerCORRECT ANSWER

Use AWS Glue transformations to attach only statement versions available at each historical decision time, retain the relevant version and publication timestamps, and repeat training and evaluation on the corrected data.

Point-in-time preparation prevents future information from entering historical examples. Retaining statement versions and publication timestamps makes the preparation auditable, and repeating training and evaluation provides more credible evidence of reliability.

MLA-C02 (beta)

A retailer uses an AWS Glue ETL job to aggregate settled transactions into store-and-product daily sales features for ML training. Each output row contains a total_sales value. An authoritative ledger contains one row per transaction with a sale_amount value. Both datasets cover the same business day, use the same currency and fixed-precision decimal representation, and contain only positive sales amounts.

A recent transformation defect omitted some transactions while producing output with valid identifiers, no nulls, and plausible sales values. The engineer must add a managed data quality check that detects whether the output's total sales differ from the ledger's total sales. The number of output rows is expected to differ from the number of ledger rows.

Which check should the engineer implement?

Show answerCORRECT ANSWER

Use an AWS Glue Data Quality AggregateMatch rule to compare SUM(total_sales) in the output with SUM(sale_amount) in the ledger and require equality.

AggregateMatch compares aggregate query results across datasets. Comparing these sums directly verifies that the aggregation preserved the authoritative sales total, even though the datasets have different granularities and row counts.

AIP-C01

A distributor maintains a vector store for a generative AI assistant that answers questions about chemical handling. A custom AWS Step Functions workflow uses AWS Lambda functions to transform approved product records into passages and update their embeddings. Each source record includes a revision, a maximum storage temperature, and its temperature unit.

A transformation defect sometimes treats Fahrenheit values as Celsius. The resulting passages are complete, contain valid numbers, and generate embeddings successfully, but state unsafe storage limits. The authoritative source values are correct. Updates are processed serially for each product, and retrieval can exclude individual products through an existing eligibility registry.

Valid updates must become searchable within 5 minutes without waiting for unrelated products. After an update finishes processing, a product must not remain searchable with either an incorrect new limit or a superseded limit. Failed records must remain available for diagnosis and reprocessing.

Which maintenance design meets these requirements?

Show answerCORRECT ANSWER

Add a Lambda validation step that independently converts the authoritative temperature into a canonical unit and compares it with the transformed passage's value and unit. Publish a revision only if they agree. On failure, mark the product ineligible for retrieval, quarantine the failed revision with its source and transformation details, and notify operators. Continue processing other products.

This checks semantic accuracy against the authoritative source rather than merely checking format or processing success. Per-product validation allows valid updates to proceed independently. Excluding a failed product prevents both incorrect replacement content and superseded content from being served, while quarantine preserves the information needed for diagnosis and reprocessing.

Practice Data Quality in a full exam

Questions on Data Quality in each practice exam, with every answer option and an explanation for each:

Start a free practice exam

What is Data Quality

3 min read

Data quality is fitness for a specific purpose, not a universal property of a dataset. A daily inventory snapshot might be adequate for demand forecasting but unsuitable for real-time order fulfillment. Engineers should define measurable acceptance criteria based on the consuming application, including the consequences of incorrect, missing, duplicated, or outdated records.

Common quality dimensions include accuracy, completeness, consistency, validity, uniqueness, and timeliness. Accuracy concerns whether a value reflects reality; validity concerns whether it follows an expected format or rule. A correctly formatted postal code can still be inaccurate for a customer's address. Similarly, a dataset with no missing values may contain fabricated defaults, duplicate customers, or stale measurements.

In cloud pipelines, assess quality during ingestion, after transformations, and before publishing data to consumers. Profile incoming datasets to understand distributions, null rates, schemas, and unexpected categories. Convert important expectations into automated checks, such as requiring transaction identifiers to be unique, rejecting impossible timestamps, and reconciling aggregate payment totals against an authoritative source. Track results over time rather than treating a successful initial assessment as permanent approval.

Use explicit failure handling. A pipeline might block publication when a required identifier is missing, quarantine malformed records for investigation, and issue a warning for a small increase in optional-field nulls. Preserve source data under appropriate access and retention controls so that corrections can be reproduced. Record dataset versions, transformation logic, ownership, and lineage to explain how a published value was produced.

AI workloads introduce additional quality concerns. Evaluate label accuracy, coverage of relevant populations and operating conditions, and separation between training and evaluation datasets. Duplicate or overlapping examples can inflate evaluation scores. A fraud dataset dominated by legitimate transactions can appear complete while providing too few useful fraud examples. Monitor production inputs for schema violations and distribution changes, but investigate changes before assuming they indicate corruption or degraded model performance.

Secure data engineering protects the quality process itself. Apply least-privilege access so ingestion systems, transformation jobs, and model developers receive only the permissions they need. Restrict writes to approved datasets, audit modifications, and validate source provenance to reduce accidental changes and data-poisoning risks. Checksums help detect unintended byte changes, while authenticated signatures can help establish origin; neither proves that the underlying facts or labels are correct. Encryption protects confidentiality but does not repair inaccurate data.

Privacy-enhancing techniques must also preserve fitness for use. Minimize collected attributes, tokenize identifiers where appropriate, and use masking or differential privacy when suitable for the use case. Tokenized data is not automatically anonymous, particularly when re-identification mappings remain available. Assess whether privacy transformations remove important predictive signals, disrupt joins, or disproportionately affect small groups.

Quality controls involve trade-offs among cost, latency, coverage, and risk. Full-dataset checks provide more coverage than sampling but can cost more and delay processing. Strict rejection rules may prevent bad data from reaching a model while also excluding legitimate unusual cases. Set risk-based thresholds with data owners, monitor exceptions, and remediate root causes rather than silently dropping every anomaly. More data is not necessarily better data, and passing automated rules does not establish fairness, security, or regulatory compliance.

Learn Data Quality in lessons

12 short lessons on Data Quality in 5 certification courses.

and 6 more lessons

Open the lessons