AWS Glue
Cloud service · AWS · Updated

AWS Glue

In short

AWS Glue is a fully managed extract, transform, and load (ETL) service that simplifies data preparation and loading for analytics for both customers and users.

Cloud Services Hub →

Free AWS Glue tools

10 key features of AWS Glue

01
ETL Capabilities

Offers both visual and code-based interfaces for data preparation and transformation, with features like crawling, schema inference, format conversion, and dataset joining.

02
AWS Glue Data Catalog

Acts as a central repository for metadata about data assets, facilitating discovery and query across AWS services like Athena, EMR, and Redshift Spectrum.

03
Serverless Execution

Enables running ETL jobs without provisioning infrastructure, where you pay only for the resources consumed during job execution.

04
Schema Registry

Validates streaming-data schemas and controls schema evolution through compatibility checks.

05
Machine Learning for Data Preparation

Incorporates machine learning techniques like data quality checks, entity extraction, and fuzzy matching to enhance data readiness for analytics.

06
Flexible Scripting

Automatically generates ETL code in Python or Scala, which users can customize and reuse, streamlining the script development process.

07
Visual ETL Tool (Glue Studio)

Provides a user-friendly interface for creating, executing, and monitoring ETL jobs, making it accessible to non-developers.

08
Automatic Schema Recognition

Detects and understands data schemas to facilitate accurate data transformation and transfer.

09
Job Scheduling and Monitoring

Offers tools for setting up job schedules and monitoring their execution, complete with logging and notifications to track ETL process efficiency.

10
Resource Usage Monitoring

Exposes resource usage as a percentage in CloudWatch and supports alarms on that usage, including workflows, triggers, jobs, concurrent job runs, blueprints, and interactive sessions.

Where AWS Glue is used

  1. 01
    Data Integration and Analytics

    AWS Glue streamlines the integration of various data sources, such as databases, data lakes, and data warehouses, offering a unified view for analytics, and supports BI initiatives by enabling comprehensive reporting and dashboard creation.

  2. 02
    ETL/ELT Workflows

    It facilitates the creation and execution of ETL (extract, transform, load) and ELT (extract, load, transform) workflows through both visual and code-based interfaces, aiding in data cleaning, normalization, and preparation.

  3. 03
    Data Lake Management and Analytics

    Glue aids in forming and analyzing data lakes on Amazon S3, working seamlessly with services like Athena, EMR, and Redshift Spectrum to analyze large datasets.

  4. 04
    Machine Learning Preparation

    Provides tools for schema inference, data quality checks, and machine learning transformations, thus readying data for machine learning applications.

  5. 05
    Application Development Support

    Simplifies data access via its data catalog, enabling applications to easily utilize analytics-ready data.

  6. 06
    Batch and Real-time Data Processing

    Supports both batch processing and real-time data integration, accommodating streaming sources like Kafka and Amazon Kinesis for continuous data analysis.

  7. 07
    Data Warehouse Modernization

    Moves, combines, replicates, and transforms data into a warehouse such as Amazon Redshift after source data is available on AWS.

  8. 08
    Log and Stream Analytics

    Processes and transforms log data and streams from various sources for real-time analytics, aiding in monitoring, troubleshooting, and insight extraction.

How AWS Glue is priced

AWS Glue uses DPUs as a measure of resource consumption for running ETL jobs and development endpoints. A DPU is a relative measure of processing power that consists of 4 vCPUs of compute capacity and 16 GB of memory. The cost is based on the number of DPUs consumed by your jobs and endpoints.

Glue ETL jobs

You are charged at a DPU-hour rate based on the DPUs used and job duration, with billing in one-second increments. AWS Glue 2.0 and later Spark ETL jobs have a one-minute minimum charge.

Glue DataBrew

DataBrew is a feature of AWS Glue that allows you to clean and normalize data without writing code. Pricing is based on the number of DataBrew nodes (virtual servers) used and the amount of time the jobs run.

Glue Interactive Sessions

Interactive sessions are billed at a DPU-hour rate according to allocated DPUs and session duration, in one-second increments with a one-minute minimum charge.

Glue Studio

Data preview sessions use 2 DPUs and are billed in one-second increments with a one-minute minimum charge. You also pay for underlying AWS services used like storage, databases etc.

Glue Data Catalog

The first one million metadata objects stored and the first one million metadata access requests each month are free. Above these allowances, storage costs $1 per 100,000 metadata objects per month, and access costs $1 per million requests. Metadata objects include tables, table versions, partitions, and databases.

Glue Data Catalog table optimization

Data Catalog table optimization, including Apache Iceberg compaction, snapshot retention, and orphan-file deletion, is billed for the DPU compute used, separately from metadata storage and access charges.

Glue Flex jobs

Flex is a lower-cost execution class for nonurgent workloads whose start and completion times can vary.

Glue Ray jobs

Ray jobs are charged per M-DPU-hour, billed per second with a one-minute minimum.

Glue development endpoints

Development endpoints are charged for provisioned time and allocated DPUs, billed per second with a ten-minute minimum; they do not time out automatically.

Glue crawlers

Crawler runs are billed by DPU usage and runtime in one-second increments, with a ten-minute minimum per crawl.

Glue DataBrew interactive sessions

DataBrew interactive usage is billed per session in 30-minute increments; the first 40 interactive sessions are free for first-time DataBrew users.

Glue Data Catalog statistics

Generating table statistics costs $0.44 per DPU-hour, billed per second with a one-minute minimum.

Glue Data Catalog materialized view refresh

Materialized view refresh costs $0.44 per DPU-hour, billed per second with a one-minute minimum.

Glue Data Quality

Data Catalog recommendation and evaluation tasks require at least 2 DPUs and have a one-minute minimum billing duration. Data quality checks embedded in ETL jobs can increase job runtime or DPU consumption.

Glue Data Quality anomaly detection

Data Catalog anomaly detection consumes one additional DPU per statistic for detection time, with a one-second minimum. Retraining also incurs DPU compute charges. Anomaly detection in Glue ETL jobs has no additional charge.

Glue zero-ETL

Application-source ingestion is charged by received data volume, with a minimum of 1 MB per ingestion request. Processing changed data written to SageMaker Lakehouse on S3 is charged by Glue DPU-hour, billed per second with a one-minute minimum; Redshift targets incur the applicable Redshift compute charges.

Services AWS Glue works with

Amazon S3Amazon S3

AWS Glue can extract, transform, and load (ETL) data to and from Amazon S3 buckets.

Amazon RDSAmazon RDS

AWS Glue can connect to and perform ETL operations on data stored in Amazon RDS databases.

Amazon KinesisAmazon Kinesis

AWS Glue can read from and write to Amazon Kinesis Data Streams and Amazon Data Firehose.

Amazon AppFlowAmazon AppFlow

AWS Glue Data Catalog can register the schema and location of data that AppFlow transfers into Amazon S3. This makes application exports discoverable as catalog tables for downstream analytics.

Amazon AuroraAmazon Aurora

AWS Glue connects to Aurora databases through JDBC connections to discover schemas and read or write data. An ETL job can extract transactional records from Aurora and transform them for a data lake.

Amazon Aurora ServerlessAmazon Aurora Serverless

AWS Glue can access Aurora Serverless databases through JDBC connections when the database endpoint is reachable from the job's network. Crawlers can discover database tables, and ETL jobs can read and write their data.

What is AWS Glue

2 min read

AWS Glue is a cloud-based service provided by Amazon Web Services (AWS) that serves as a fully managed extract, transform, and load (ETL) solution. It is designed to streamline the process of preparing and combining data for analytics and data processing workflows.

AWS Glue automates the time-consuming steps of data preparation, such as discovering, categorizing, cleaning, enriching, and moving data between various data stores. Its Data Catalog is a centralized technical metadata repository, with AWS Lake Formation capabilities for centralized data governance. This catalog provides a unified view of all data sources and simplifies the management of data across different AWS services and your data warehouse.

AWS Glue Data Quality can recommend rules for data lakes, evaluate rules in pipelines, and calculate quality scores visible in the Data Catalog. Rules can be customized and scheduled, with actions to alert teams when quality issues occur.

AWS Glue dynamically generates ETL code in Python or Scala, which can be further customized and stored for future use. It is serverless, meaning that it automatically provisions the compute resources required to process data according to the workload. This makes it a scalable solution that adjusts to the volume of data, whether it’s processing a few records or terabytes of data. Glue is integrated with a wide range of AWS services, making it a versatile tool for different ETL tasks and enabling seamless data integration across the AWS ecosystem.

AWS Glue has standard and flexible execution classes. Standard execution is intended for time-sensitive workloads requiring fast startup and dedicated resources; Flex suits nonurgent jobs whose start and completion times may vary.

For interactive development, Glue Interactive Sessions let you explore, experiment on, and process data using your preferred IDE or notebook.

Custom readers, writers, and transformations can be imported into Glue ETL jobs as libraries.

Glue Zero-ETL defaults to 25 integrations per account, with up to 15 integrations associated with each source and up to 50 associated with each target.

In addition to ETL capabilities, AWS Glue offers features like the Glue DataBrew, which allows users to clean and normalize data without writing code. It provides a visual interface to easily combine, transform, and categorize data. AWS Glue's ability to handle both batch and streaming data makes it suitable for various data processing needs, from big data analytics to real-time analytics applications. 

For streaming sources, Data Catalog tables and data stream properties are defined manually rather than populated by a data-store crawler.

This comprehensive service simplifies data preparation and integration, facilitating more efficient and effective data analysis and decision-making processes within organizations.

Concepts related to AWS Glue

AWS architecture icons are the property of Amazon Web Services. BlowStack is not affiliated with or endorsed by AWS.