CLOUD SERVICE · AWS

AWS Glue DataBrew - AWS Glue DataBrew is a visual data preparation service for profiling, cleaning, and transforming datasets using reusable recipes.

DataBrew provides an interactive workspace for inspecting data samples and defining transformations without writing transformation code. Managed recipe and profile jobs apply preparation steps, generate statistics, and evaluate data quality rules for recurring analytics and machine learning workflows.

Cloud Services Hub →

What is AWS Glue DataBrew

Read the extensive description

AWS Glue DataBrew supports visual data preparation through datasets, projects, recipes, and jobs. A dataset identifies an input source and its format or connection configuration. A project provides an interactive workspace for examining a dataset sample and building transformations. A recipe records those transformations as ordered steps, while jobs execute recipes or generate data profiles independently of an interactive session.

The project workspace operates on a sample rather than displaying every record in the source. Engineers can inspect column types, examine values, and preview operations such as type conversion, filtering, deduplication, and text normalization. Sample-based development reduces the amount of data needed for exploration, but rare values and production-scale problems may not appear in the preview. Validate recipes against representative inputs and review job results before publishing prepared data to consumers.

Recipes provide a reusable definition of preparation logic. Supported operations include handling missing values, splitting and combining columns, changing formats, joining datasets, and aggregating records. Publish recipe versions and configure production jobs to use an intended version so that experimental changes do not unexpectedly alter recurring processing. Retain the input location, recipe version, and job configuration needed to reproduce important outputs.

Profile jobs calculate dataset and column statistics, including missing values, distinct values, and distributions. Rulesets define data quality requirements that can be evaluated during profiling, such as completeness thresholds or permitted numeric ranges. Profiling and recipe execution are separate activities: successful transformation does not by itself establish that the result meets business quality requirements. Build explicit checks and failure-handling logic into workflows that require quality gates.

DataBrew reads from Amazon S3, supported AWS Glue Data Catalog tables, and supported database connections. Recipe jobs can write prepared files to S3 and support AWS Glue Data Catalog output configurations, including creating a new output table. Choose the output configuration according to how downstream systems discover and consume the data. When using S3-only output, register or update catalog metadata separately if catalog-based consumers require it; a separate crawler is not inherently required when the job uses a supported catalog output configuration.

Operational setup includes an IAM execution role with permissions for inputs, outputs, catalog resources, logging, and applicable encryption keys. Database sources also require credentials and network connectivity. Schedule jobs for recurring preparation or orchestrate them with a workflow service when consumers must wait for successful completion. Keep raw data separate from prepared output, plan for repeated runs and output retention, and inspect execution logs when jobs fail. DataBrew fits visual, batch-oriented preparation; workloads requiring streaming execution, specialized libraries, or unsupported transformations need a different processing tool.

Key AWS Glue DataBrew Features

Visual preparation workspace

Inspect a dataset sample and preview transformations in an interactive project. Choose representative sampling settings and test full-job behavior before promoting a recipe to production.

Reusable, versioned recipes

Store ordered transformation steps in recipes and publish versions for repeatable execution. Configure production jobs with the intended published version to separate development changes from scheduled processing.

Managed recipe execution

Run preparation jobs without provisioning processing servers. Configure the recipe, input, output, and processing capacity, then monitor execution status and results.

Dataset profiling

Generate dataset and column statistics to investigate missing values, cardinality, and value distributions. Use profile reports to identify issues that may not appear in the interactive sample.

Data quality rulesets

Define validation rules and evaluate them during profile jobs. For example, check that required identifiers meet a completeness threshold or that transaction amounts fall within an accepted range.

Built-in transformations

Apply supported operations for type conversion, deduplication, filtering, text normalization, joins, and aggregation. Review schema changes and record counts to detect unintended effects.

Sensitive-data handling

Use supported sensitive-data detection and transformation capabilities to identify and protect selected fields. Validate the transformed output and restrict access to raw datasets and profiling reports.

Catalog-aware outputs

Configure supported recipe-job outputs through the AWS Glue Data Catalog, including creating a new output table. For S3-only outputs, manage catalog registration separately when downstream consumers need table metadata.

Scheduling and automation

Schedule recurring jobs and manage resources through DataBrew APIs. Use workflow orchestration when downstream tasks must wait for job completion or respond to failures.

AWS Glue DataBrew Use Cases

Prepare data lake datasets

Normalize incoming files, correct column types, and write prepared output to a separate S3 location. Configure catalog output when consumers need a discoverable table, and preserve raw inputs for reprocessing.

Assess incoming data quality

Profile new sources and recurring deliveries before using them in analytics. Evaluate rules for completeness and valid ranges, and have the surrounding workflow act on validation results.

Standardize reporting inputs

Apply consistent date formats, category values, and text cleanup across recurring extracts. Publish recipe versions so reporting teams can reproduce the preparation logic used for each release.

Prepare machine learning data

Handle missing values, remove duplicates, and standardize supported fields before downstream feature engineering or training. Validate that transformations preserve labels and do not unintentionally discard important records.

Prepare restricted datasets for sharing

Transform selected sensitive fields before making prepared data available to a broader audience. Combine masking or other supported transformations with access controls on source data, outputs, and profile reports.

AWS Glue DataBrew pricing models

Interactive sessions

Interactive project usage is charged in 30-minute session units. Budget for exploration and recipe development separately from automated job execution, using the applicable regional rates.

Job processing

Recipe and profile jobs are billed according to processing node usage and execution duration. Configured capacity, data volume, and transformation complexity affect total processing cost.

Supporting AWS services

Additional charges can apply for S3 storage and requests, database operations, logging, encryption-key usage, and data transfer. Include prepared datasets, retained profile reports, and repeated processing in cost estimates.

Services AWS Glue DataBrew integrates with

A common architecture uses Amazon S3 for raw inputs, prepared outputs, and profiling reports. DataBrew recipe jobs transform incoming files into a format and schema suitable for downstream analytics. Keep raw and prepared locations separate, and define how consumers identify the output from each successful run.The AWS Glue Data Catalog provides table metadata for supported inputs and outputs. A recipe job can use a supported catalog output configuration, including creating a new output table, rather than requiring a separate crawler in every workflow. If the job writes only to S3, register or update metadata separately when downstream tools need a catalog table. For Redshift-sourced workflows, configure a supported connection and validate credentials, network access, and extraction permissions before scheduling recurring runs.Use AWS Step Functions to coordinate preparation with dependent tasks and wait for successful job completion. Where data quality determines whether processing continues, include profiling and explicit evaluation of its results in the workflow. CloudWatch logs support failure investigation. Apply least-privilege IAM permissions and applicable encryption-key policies across sources, outputs, catalog resources, and reports, and keep production recipe versions separate from interactive development.

AWS Glue

DataBrew can read supported catalog tables and use supported Data Catalog output configurations for recipe jobs, including creating a new output table. For example, prepare an S3-backed sales dataset and configure the job to publish its result through a catalog table for downstream analytics. If the job instead writes only to S3, add a separate metadata-registration or crawler step when catalog-based consumers require one.

Open AWS Glue →

Amazon Redshift

DataBrew supports Amazon Redshift as a database source through an appropriately configured connection. For example, extract customer records from a warehouse, standardize selected fields, and write the prepared dataset to S3. Configure database credentials, network reachability, and read permissions, and account for extraction load on the warehouse.

Open Amazon Redshift →

AWS Step Functions

Step Functions provides an optimized integration that starts a DataBrew job and waits for completion. For example, run a preparation recipe before launching a downstream task that consumes the output dataset. Add retry and failure-handling paths, and grant the workflow role permission to start and monitor the job.

Open AWS Step Functions →

Amazon CloudWatch

CloudWatch logs support troubleshooting DataBrew job execution. For example, inspect logs from a failed recipe job to determine whether the cause was inaccessible input data or a transformation error. Give the execution role the necessary logging permissions and set retention policies appropriate for operational and compliance needs.

Open Amazon CloudWatch →

Amazon S3

Amazon S3 stores source datasets, prepared output, and profile reports for common DataBrew workflows. For example, read daily CSV deliveries from a raw-data prefix and write cleaned Parquet files to a separate prepared-data prefix. Grant the execution role the required object access and any applicable encryption-key permissions.

Open Amazon S3 →