Cloud service · AWS · Updated

AWS Glue DataBrew - AWS Glue DataBrew is a visual data preparation service for profiling, cleaning, and transforming datasets using reusable recipes.

In short

DataBrew provides an interactive workspace for inspecting data samples and defining transformations without writing transformation code. Managed recipe and profile jobs apply preparation steps, generate statistics, and evaluate data quality rules for recurring analytics and machine learning workflows.

Cloud Services Hub →

9 key features of AWS Glue DataBrew

01
Visual preparation workspace

Inspect a dataset sample and preview transformations in an interactive project. Choose representative sampling settings and test full-job behavior before promoting a recipe to production.

02
Reusable, versioned recipes

Store ordered transformation steps in recipes and publish versions for repeatable execution. Configure production jobs with the intended published version to separate development changes from scheduled processing.

03
Managed recipe execution

Run preparation jobs without provisioning processing servers. Configure the recipe, input, output, and processing capacity, then monitor execution status and results.

04
Dataset profiling

Generate dataset and column statistics to investigate missing values, cardinality, and value distributions. Use profile reports to identify issues that may not appear in the interactive sample.

05
Data quality rulesets

Define validation rules and evaluate them during profile jobs. For example, check that required identifiers meet a completeness threshold or that transaction amounts fall within an accepted range.

06
Built-in transformations

Apply supported operations for type conversion, deduplication, filtering, text normalization, joins, and aggregation. Review schema changes and record counts to detect unintended effects.

07
Sensitive-data handling

Use supported sensitive-data detection and transformation capabilities to identify and protect selected fields. Validate the transformed output and restrict access to raw datasets and profiling reports.

08
Catalog-aware outputs

Configure supported recipe-job outputs through the AWS Glue Data Catalog, including creating a new output table. For S3-only outputs, manage catalog registration separately when downstream consumers need table metadata.

09
Scheduling and automation

Schedule recurring jobs and manage resources through DataBrew APIs. Use workflow orchestration when downstream tasks must wait for job completion or respond to failures.

Where AWS Glue DataBrew is used

  1. 01
    Prepare data lake datasets

    Normalize incoming files, correct column types, and write prepared output to a separate S3 location. Configure catalog output when consumers need a discoverable table, and preserve raw inputs for reprocessing.

  2. 02
    Assess incoming data quality

    Profile new sources and recurring deliveries before using them in analytics. Evaluate rules for completeness and valid ranges, and have the surrounding workflow act on validation results.

  3. 03
    Standardize reporting inputs

    Apply consistent date formats, category values, and text cleanup across recurring extracts. Publish recipe versions so reporting teams can reproduce the preparation logic used for each release.

  4. 04
    Prepare machine learning data

    Handle missing values, remove duplicates, and standardize supported fields before downstream feature engineering or training. Validate that transformations preserve labels and do not unintentionally discard important records.

  5. 05
    Prepare restricted datasets for sharing

    Transform selected sensitive fields before making prepared data available to a broader audience. Combine masking or other supported transformations with access controls on source data, outputs, and profile reports.

How AWS Glue DataBrew is priced

Interactive sessions

Interactive project usage is charged in 30-minute session units. Budget for exploration and recipe development separately from automated job execution, using the applicable regional rates.

Job processing

Recipe and profile jobs are billed according to processing node usage and execution duration. Configured capacity, data volume, and transformation complexity affect total processing cost.

Supporting AWS services

Additional charges can apply for S3 storage and requests, database operations, logging, encryption-key usage, and data transfer. Include prepared datasets, retained profile reports, and repeated processing in cost estimates.

Services AWS Glue DataBrew works with

A common architecture uses Amazon S3 for raw inputs, prepared outputs, and profiling reports. DataBrew recipe jobs transform incoming files into a format and schema suitable for downstream analytics. Keep raw and prepared locations separate, and define how consumers identify the output from each successful run.

The AWS Glue Data Catalog provides table metadata for supported inputs and outputs. A recipe job can use a supported catalog output configuration, including creating a new output table, rather than requiring a separate crawler in every workflow. If the job writes only to S3, register or update metadata separately when downstream tools need a catalog table. For Redshift-sourced workflows, configure a supported connection and validate credentials, network access, and extraction permissions before scheduling recurring runs.

Use AWS Step Functions to coordinate preparation with dependent tasks and wait for successful job completion. Where data quality determines whether processing continues, include profiling and explicit evaluation of its results in the workflow. CloudWatch logs support failure investigation. Apply least-privilege IAM permissions and applicable encryption-key policies across sources, outputs, catalog resources, and reports, and keep production recipe versions separate from interactive development.

Amazon S3Amazon S3

Amazon S3 stores source datasets, prepared output, and profile reports for common DataBrew workflows. For example, read daily CSV deliveries from a raw-data prefix and write cleaned Parquet files to a separate prepared-data prefix. Grant the execution role the required object access and any applicable encryption-key permissions.

AWS GlueAWS Glue

DataBrew can read supported catalog tables and use supported Data Catalog output configurations for recipe jobs, including creating a new output table. For example, prepare an S3-backed sales dataset and configure the job to publish its result through a catalog table for downstream analytics. If the job instead writes only to S3, add a separate metadata-registration or crawler step when catalog-based consumers require one.

Amazon RedshiftAmazon Redshift

DataBrew supports Amazon Redshift as a database source through an appropriately configured connection. For example, extract customer records from a warehouse, standardize selected fields, and write the prepared dataset to S3. Configure database credentials, network reachability, and read permissions, and account for extraction load on the warehouse.

Amazon CloudWatchAmazon CloudWatch

CloudWatch logs support troubleshooting DataBrew job execution. For example, inspect logs from a failed recipe job to determine whether the cause was inaccessible input data or a transformation error. Give the execution role the necessary logging permissions and set retention policies appropriate for operational and compliance needs.

AWS Step FunctionsAWS Step Functions

Step Functions provides an optimized integration that starts a DataBrew job and waits for completion. For example, run a preparation recipe before launching a downstream task that consumes the output dataset. Add retry and failure-handling paths, and grant the workflow role permission to start and monitor the job.

Amazon AppFlowAmazon AppFlow

AWS Glue DataBrew can use data delivered by Amazon AppFlow flows as dataset inputs for profiling and recipe jobs. AppFlow is supported as a data source, not as an output destination for DataBrew recipe jobs.

Amazon AuroraAmazon Aurora

AWS Glue DataBrew can connect to Aurora MySQL and Aurora PostgreSQL tables through AWS Glue Data Catalog connections. This lets you profile and clean database data using reusable recipes.

Amazon Aurora ServerlessAmazon Aurora Serverless

AWS Glue DataBrew can access Aurora Serverless databases through compatible MySQL or PostgreSQL connections registered in the AWS Glue Data Catalog. The connection must provide network access to the database endpoint.

Amazon EC2Amazon EC2

AWS Glue DataBrew uses Amazon EC2 networking APIs to create and manage network interfaces for connections to data sources in a VPC. Its service role needs the corresponding permissions to establish that connectivity.

Amazon EventBridgeAmazon EventBridge

AWS Glue DataBrew sends job-state events to Amazon EventBridge. EventBridge rules can react to completed or failed jobs, for example by sending a notification or starting a downstream workflow.

AWS Glue DataBrew exam questions

Real practice exam questions on AWS Glue DataBrew, with the correct answer and its explanation. Answer them with every option in the exam simulator.

AIF-C01

An agricultural company is preparing crop measurements from several research partners to train a yield-prediction model. The partners use different measurement units and missing-value conventions. Before training, analysts must prepare the data without writing transformation code. A governance reviewer must also be able to verify which approved sources contributed to the training dataset and reproduce the preparation steps. Which approach best meets these requirements?

Show answerCORRECT ANSWER

Use AWS Glue DataBrew recipes to standardize units and missing values. Retain source identifiers linked to approval records, preserve the input versions, and record the published recipe version and job configuration used for each output.

DataBrew provides visual, reusable preparation steps without transformation code. Linking source identifiers to approval records supports source verification, while retaining the inputs, recipe version, and job configuration supports traceability and reproducibility.

MLA-C02 (beta)

A logistics company receives daily shipment CSV files in Amazon S3. Analysts need a visual, no-code workflow to trim whitespace and convert a shipping_cost column from text to a numeric type. The original files must remain unchanged. Before the prepared data becomes available for ML training, the company must verify that every output record has a non-null, nonnegative shipping_cost. Invalid values are rare and might not appear in an interactive preview. A failed validation must block publication of the entire daily dataset.

Which solution meets these requirements?

Show answerCORRECT ANSWER

Create an AWS Glue DataBrew recipe for the transformations and write its job output to a staging prefix. Run a profile job on the staged output with full-dataset sampling and a ruleset requiring 100% non-null and nonnegative shipping_cost values. Configure workflow logic to publish the output only when profiling completes and all rules pass.

A recipe job performs the visual transformations without changing the source files. Profiling the complete staged output checks the actual transformed values, including rare invalid records. Explicit workflow logic that requires completed profiling and passing rules prevents failed or incomplete validation from publishing data.

DEA-C01

A manufacturing company stores equipment measurements in Amazon S3. Each row contains readings from 20 numeric sensors taken at the same instant. Before selecting inputs for an analytics model, a data engineer must visually identify sensor pairs with strong linear relationships, including pairs where one reading decreases as the other increases. The assessment must use the original, unaggregated measurements and must not require custom analysis code.

Which solution best meets these requirements?

Show answerCORRECT ANSWER

Run an AWS Glue DataBrew profile job with column correlations enabled for the sensor columns. Review the column-correlation results in the console and identify pairs with correlation coefficients close to either +1 or -1.

DataBrew profiling can calculate correlations between numeric columns and display the results for inspection. Coefficients close to +1 indicate strong positive linear relationships, while coefficients close to -1 indicate strong inverse linear relationships. This evaluates relationships between the original sensor measurements without custom code.

Practice AWS Glue DataBrew in a full exam

Questions on AWS Glue DataBrew in each practice exam, with every answer option and an explanation for each:

Start a free practice exam

What is AWS Glue DataBrew

2 min read

AWS Glue DataBrew supports visual data preparation through datasets, projects, recipes, and jobs. A dataset identifies an input source and its format or connection configuration. A project provides an interactive workspace for examining a dataset sample and building transformations. A recipe records those transformations as ordered steps, while jobs execute recipes or generate data profiles independently of an interactive session.

A DataBrew dataset provides read-only access to its source data.

The project workspace operates on a sample rather than displaying every record in the source. Engineers can inspect column types, examine values, and preview operations such as type conversion, filtering, deduplication, and text normalization. Sample-based development reduces the amount of data needed for exploration, but rare values and production-scale problems may not appear in the preview. Validate recipes against representative inputs and review job results before publishing prepared data to consumers.

By default, the project sample contains the first 500 rows of the dataset.

Recipes provide a reusable definition of preparation logic. Supported operations include handling missing values, splitting and combining columns, changing formats, joining datasets, and aggregating records. Publish recipe versions and configure production jobs to use an intended version so that experimental changes do not unexpectedly alter recurring processing. Retain the input location, recipe version, and job configuration needed to reproduce important outputs.

The work-in-progress recipe is saved automatically, allowing you to leave the project and resume editing later.

Profile jobs calculate dataset and column statistics, including missing values, distinct values, and distributions. Rulesets define data quality requirements that can be evaluated during profiling, such as completeness thresholds or permitted numeric ranges. Profiling and recipe execution are separate activities: successful transformation does not by itself establish that the result meets business quality requirements. Build explicit checks and failure-handling logic into workflows that require quality gates.

DataBrew reads from Amazon S3, supported AWS Glue Data Catalog tables, and supported database connections. Recipe jobs can write prepared files to S3 and support AWS Glue Data Catalog output configurations, including creating a new output table. Choose the output configuration according to how downstream systems discover and consume the data. When using S3-only output, register or update catalog metadata separately if catalog-based consumers require it; a separate crawler is not inherently required when the job uses a supported catalog output configuration.

Operational setup includes an IAM execution role with permissions for inputs, outputs, catalog resources, logging, and applicable encryption keys. Database sources also require credentials and network connectivity. Schedule jobs for recurring preparation or orchestrate them with a workflow service when consumers must wait for successful completion. Keep raw data separate from prepared output, plan for repeated runs and output retention, and inspect execution logs when jobs fail. DataBrew fits visual, batch-oriented preparation; workloads requiring streaming execution, specialized libraries, or unsupported transformations need a different processing tool.

Learn AWS Glue DataBrew in lessons

8 short lessons on AWS Glue DataBrew in 3 certification courses.

and 2 more lessons

Open the lessons

Concepts related to AWS Glue DataBrew

AWS architecture icons are the property of Amazon Web Services. BlowStack is not affiliated with or endorsed by AWS.