# AWS Glue

Cloud service · AWS. Updated 2026-11-12. Page: https://blowstack.com/tools/cloud-services-hub/aws-glue

> AWS Glue is a fully managed extract, transform, and load (ETL) service that simplifies data preparation and loading for analytics for both customers and users.

Practice exams covering AWS Glue: [AIB-C01 (beta) AWS Certified AI Business Strategist](https://blowstack.com/practice-exams/aws-certified-ai-business-strategist), [AIF-C01 AWS Certified AI Practitioner](https://blowstack.com/practice-exams/aws-certified-ai-practitioner), [CLF-C02 AWS Certified Cloud Practitioner](https://blowstack.com/practice-exams/aws-certified-cloud-practitioner), [DEA-C01 AWS Certified Data Engineer - Associate](https://blowstack.com/practice-exams/aws-certified-data-engineer-associate), [DOP-C02 AWS Certified DevOps Engineer - Professional](https://blowstack.com/practice-exams/aws-certified-devops-engineer-professional), [AIP-C01 AWS Certified Generative AI Developer - Professional](https://blowstack.com/practice-exams/aws-certified-generative-ai-developer-professional), [MLA-C02 (beta) AWS Certified Machine Learning Engineer - Associate](https://blowstack.com/practice-exams/aws-certified-machine-learning-engineer-associate), [SAA-C03 AWS Certified Solutions Architect - Associate](https://blowstack.com/practice-exams/aws-certified-solutions-architect-associate), [SAP-C02 AWS Certified Solutions Architect - Professional](https://blowstack.com/practice-exams/aws-certified-solutions-architect-professional).

## 10 key features of AWS Glue

1. **ETL Capabilities**: Offers both visual and code-based interfaces for data preparation and transformation, with features like crawling, schema inference, format conversion, and dataset joining.
2. **AWS Glue Data Catalog**: Acts as a central repository for metadata about data assets, facilitating discovery and query across AWS services like Athena, EMR, and Redshift Spectrum.
3. **Serverless Execution**: Enables running ETL jobs without provisioning infrastructure, where you pay only for the resources consumed during job execution.
4. **Schema Registry**: Validates streaming-data schemas and controls schema evolution through compatibility checks.
5. **Machine Learning for Data Preparation**: Incorporates machine learning techniques like data quality checks, entity extraction, and fuzzy matching to enhance data readiness for analytics.
6. **Flexible Scripting**: Automatically generates ETL code in Python or Scala, which users can customize and reuse, streamlining the script development process.
7. **Visual ETL Tool (Glue Studio)**: Provides a user-friendly interface for creating, executing, and monitoring ETL jobs, making it accessible to non-developers.
8. **Automatic Schema Recognition**: Detects and understands data schemas to facilitate accurate data transformation and transfer.
9. **Job Scheduling and Monitoring**: Offers tools for setting up job schedules and monitoring their execution, complete with logging and notifications to track ETL process efficiency.
10. **Resource Usage Monitoring**: Exposes resource usage as a percentage in CloudWatch and supports alarms on that usage, including workflows, triggers, jobs, concurrent job runs, blueprints, and interactive sessions.

## Where AWS Glue is used

1. **Data Integration and Analytics**: AWS Glue streamlines the integration of various data sources, such as databases, data lakes, and data warehouses, offering a unified view for analytics, and supports BI initiatives by enabling comprehensive reporting and dashboard creation.
2. **ETL/ELT Workflows**: It facilitates the creation and execution of ETL (extract, transform, load) and ELT (extract, load, transform) workflows through both visual and code-based interfaces, aiding in data cleaning, normalization, and preparation.
3. **Data Lake Management and Analytics**: Glue aids in forming and analyzing data lakes on Amazon S3, working seamlessly with services like Athena, EMR, and Redshift Spectrum to analyze large datasets.
4. **Machine Learning Preparation**: Provides tools for schema inference, data quality checks, and machine learning transformations, thus readying data for machine learning applications.
5. **Application Development Support**: Simplifies data access via its data catalog, enabling applications to easily utilize analytics-ready data.
6. **Batch and Real-time Data Processing**: Supports both batch processing and real-time data integration, accommodating streaming sources like Kafka and Amazon Kinesis for continuous data analysis.
7. **Data Warehouse Modernization**: Moves, combines, replicates, and transforms data into a warehouse such as Amazon Redshift after source data is available on AWS.
8. **Log and Stream Analytics**: Processes and transforms log data and streams from various sources for real-time analytics, aiding in monitoring, troubleshooting, and insight extraction.

## How AWS Glue is priced

AWS Glue uses DPUs as a measure of resource consumption for running ETL jobs and development endpoints. A DPU is a relative measure of processing power that consists of 4 vCPUs of compute capacity and 16 GB of memory. The cost is based on the number of DPUs consumed by your jobs and endpoints.

- **Glue ETL jobs**: You are charged at a DPU-hour rate based on the DPUs used and job duration, with billing in one-second increments. AWS Glue 2.0 and later Spark ETL jobs have a one-minute minimum charge.
- **Glue DataBrew**: DataBrew is a feature of AWS Glue that allows you to clean and normalize data without writing code. Pricing is based on the number of DataBrew nodes (virtual servers) used and the amount of time the jobs run.
- **Glue Interactive Sessions**: Interactive sessions are billed at a DPU-hour rate according to allocated DPUs and session duration, in one-second increments with a one-minute minimum charge.
- **Glue Studio**: Data preview sessions use 2 DPUs and are billed in one-second increments with a one-minute minimum charge. You also pay for underlying AWS services used like storage, databases etc.
- **Glue Data Catalog**: The first one million metadata objects stored and the first one million metadata access requests each month are free. Above these allowances, storage costs $1 per 100,000 metadata objects per month, and access costs $1 per million requests. Metadata objects include tables, table versions, partitions, and databases.
- **Glue Data Catalog table optimization**: Data Catalog table optimization, including Apache Iceberg compaction, snapshot retention, and orphan-file deletion, is billed for the DPU compute used, separately from metadata storage and access charges.
- **Glue Flex jobs**: Flex is a lower-cost execution class for nonurgent workloads whose start and completion times can vary.
- **Glue Ray jobs**: Ray jobs are charged per M-DPU-hour, billed per second with a one-minute minimum.
- **Glue development endpoints**: Development endpoints are charged for provisioned time and allocated DPUs, billed per second with a ten-minute minimum; they do not time out automatically.
- **Glue crawlers**: Crawler runs are billed by DPU usage and runtime in one-second increments, with a ten-minute minimum per crawl.
- **Glue DataBrew interactive sessions**: DataBrew interactive usage is billed per session in 30-minute increments; the first 40 interactive sessions are free for first-time DataBrew users.
- **Glue Data Catalog statistics**: Generating table statistics costs $0.44 per DPU-hour, billed per second with a one-minute minimum.
- **Glue Data Catalog materialized view refresh**: Materialized view refresh costs $0.44 per DPU-hour, billed per second with a one-minute minimum.
- **Glue Data Quality**: Data Catalog recommendation and evaluation tasks require at least 2 DPUs and have a one-minute minimum billing duration. Data quality checks embedded in ETL jobs can increase job runtime or DPU consumption.
- **Glue Data Quality anomaly detection**: Data Catalog anomaly detection consumes one additional DPU per statistic for detection time, with a one-second minimum. Retraining also incurs DPU compute charges. Anomaly detection in Glue ETL jobs has no additional charge.
- **Glue zero-ETL**: Application-source ingestion is charged by received data volume, with a minimum of 1 MB per ingestion request. Processing changed data written to SageMaker Lakehouse on S3 is charged by Glue DPU-hour, billed per second with a one-minute minimum; Redshift targets incur the applicable Redshift compute charges.

## Services AWS Glue works with

- **[Amazon S3](https://blowstack.com/tools/cloud-services-hub/amazon-s3)**: AWS Glue can extract, transform, and load (ETL) data to and from Amazon S3 buckets.
- **[Amazon RDS](https://blowstack.com/tools/cloud-services-hub/amazon-rds)**: AWS Glue can connect to and perform ETL operations on data stored in Amazon RDS databases.
- **[Amazon Redshift](https://blowstack.com/tools/cloud-services-hub/amazon-redshift)**: AWS Glue can load data into and extract data from Amazon Redshift data warehouses.
- **[Amazon Athena](https://blowstack.com/tools/cloud-services-hub/amazon-athena)**: AWS Glue can catalog data that can be queried directly with Amazon Athena.
- **[AWS Lake Formation](https://blowstack.com/tools/cloud-services-hub/aws-lake-formation)**: AWS Glue Data Catalog is used by AWS Lake Formation to manage and secure data lakes.
- **[Amazon DynamoDB](https://blowstack.com/tools/cloud-services-hub/amazon-dynamodb)**: AWS Glue can connect to and perform ETL operations on data from Amazon DynamoDB tables.
- **[Amazon Kinesis](https://blowstack.com/tools/cloud-services-hub/amazon-kinesis)**: AWS Glue can read from and write to Amazon Kinesis Data Streams and Amazon Data Firehose.
- **[Amazon AppFlow](https://blowstack.com/tools/cloud-services-hub/amazon-appflow)**: AWS Glue Data Catalog can register the schema and location of data that AppFlow transfers into Amazon S3. This makes application exports discoverable as catalog tables for downstream analytics.
- **[Amazon Aurora](https://blowstack.com/tools/cloud-services-hub/amazon-aurora)**: AWS Glue connects to Aurora databases through JDBC connections to discover schemas and read or write data. An ETL job can extract transactional records from Aurora and transform them for a data lake.
- **[Amazon Aurora Serverless](https://blowstack.com/tools/cloud-services-hub/amazon-aurora-serverless)**: AWS Glue can access Aurora Serverless databases through JDBC connections when the database endpoint is reachable from the job's network. Crawlers can discover database tables, and ETL jobs can read and write their data.
- **[Amazon Bedrock](https://blowstack.com/tools/cloud-services-hub/amazon-bedrock)**: AWS Glue Data Catalog can supply table metadata for structured-data knowledge bases in Amazon Bedrock that use Amazon Redshift as their query engine. This lets a knowledge base answer natural-language questions over supported cataloged lake data.
- **[Amazon CloudWatch](https://blowstack.com/tools/cloud-services-hub/amazon-cloudwatch)**: AWS Glue publishes job logs and operational metrics to CloudWatch. You can inspect Spark execution logs, monitor resource utilization, and configure alarms for processing problems.
- **[Amazon Connect](https://blowstack.com/tools/cloud-services-hub/amazon-connect)**: AWS Glue Data Catalog exposes tables for data shared through the Amazon Connect analytics data lake. These tables describe contact-center datasets that authorized analytics tools can query.
- **[Amazon Data Firehose](https://blowstack.com/tools/cloud-services-hub/amazon-kinesis-data-firehose)**: AWS Glue Data Catalog supplies the table schema that Data Firehose uses when converting JSON records to Parquet or ORC before delivery to Amazon S3. Schema versions in the catalog control how incoming records are interpreted.
- **[Amazon DataZone](https://blowstack.com/tools/cloud-services-hub/amazon-datazone)**: AWS Glue databases and tables can be ingested into DataZone through an AWS Glue data source. DataZone adds business metadata and publishing workflows to those technical assets so users can discover and request access to them.
- **[Amazon DocumentDB (with MongoDB compatibility)](https://blowstack.com/tools/cloud-services-hub/amazon-documentdb-(with-mongodb-compatibility))**: AWS Glue provides connections for reading and writing Amazon DocumentDB collections. ETL jobs can transform document data into tabular datasets or load processed records back into DocumentDB.
- **[Amazon EC2](https://blowstack.com/tools/cloud-services-hub/amazon-ec2)**: AWS Glue development endpoints support notebook servers hosted on EC2 for interactive ETL development. The notebook server connects to the development endpoint to execute and test Glue code.
- **[Amazon EMR](https://blowstack.com/tools/cloud-services-hub/amazon-emr)**: AWS Glue Data Catalog can serve as the external Hive metastore for supported applications on EMR. Spark and Hive workloads can use shared table definitions instead of maintaining a separate metastore.
- **[Amazon EventBridge](https://blowstack.com/tools/cloud-services-hub/amazon-eventbridge)**: AWS Glue emits job, crawler, and Data Catalog events to EventBridge. EventBridge can also deliver events to a Glue workflow trigger, allowing workflows to start when matching events arrive.
- **[Amazon HealthLake](https://blowstack.com/tools/cloud-services-hub/amazon-healthlake)**: AWS Glue Data Catalog holds table metadata used by HealthLake's analytics integration. Together with Lake Formation permissions, this makes supported HealthLake datasets available for SQL analysis.
- **[Amazon Keyspaces (for Apache Cassandra)](https://blowstack.com/tools/cloud-services-hub/amazon-keyspaces-(for-apache-cassandra))**: AWS Glue Spark jobs can use the Amazon Keyspaces connector to read and write Cassandra-compatible tables. This supports tasks such as loading transformed S3 data into Keyspaces.
- **[Amazon Kinesis Data Streams](https://blowstack.com/tools/cloud-services-hub/amazon-kinesis-data-streams)**: AWS Glue streaming ETL jobs can consume records from Kinesis Data Streams and continuously transform them into destination datasets. Glue can also use its Schema Registry to serialize and validate records produced to or consumed from streams.
- **[Amazon Managed Service for Apache Flink](https://blowstack.com/tools/cloud-services-hub/amazon-managed-service-for-apache-flink)**: AWS Glue Schema Registry integrates with Flink applications through serialization and deserialization libraries. Applications can validate streaming records against registered schemas and manage schema evolution.
- **[Amazon Managed Streaming for Apache Kafka](https://blowstack.com/tools/cloud-services-hub/amazon-managed-streaming-for-apache-kafka)**: AWS Glue streaming ETL jobs can consume Kafka topics hosted on Amazon MSK. AWS Glue Schema Registry also provides schema management and compatible serializers for Kafka producers and consumers.
- **[Amazon Managed Workflows for Apache Airflow (Amazon MWAA)](https://blowstack.com/tools/cloud-services-hub/amazon-managed-workflows-for-apache-airflow-(amazon-mwaa))**: AWS Glue jobs and crawlers can be orchestrated from MWAA using Airflow's AWS Glue operators and sensors. A DAG can start a job, wait for completion, and then run dependent processing tasks.
- **[Amazon Neptune](https://blowstack.com/tools/cloud-services-hub/amazon-neptune)**: AWS Glue jobs can use the Neptune connector to read and write graph data. This allows ETL pipelines to transform source records into vertices and edges or process data extracted from a graph.
- **[Amazon OpenSearch Service](https://blowstack.com/tools/cloud-services-hub/amazon-opensearch-service)**: AWS Glue for Spark provides an OpenSearch Service connection for reading and writing indexed data. Jobs can clean and enrich records before indexing them or extract indexed documents for further processing.
- **[Amazon Q](https://blowstack.com/tools/cloud-services-hub/amazon-q)**: AWS Glue Studio integrates Amazon Q Developer to help author and troubleshoot data integration jobs. You can describe a transformation in natural language to generate code or request explanations of Glue functionality and job errors.
- **[Amazon QuickSight](https://blowstack.com/tools/cloud-services-hub/amazon-quicksight)**: AWS Glue Data Catalog tables can be used in QuickSight datasets through an Athena data source. Glue supplies the schemas and partition metadata for the underlying lake data that QuickSight queries.
- **[Amazon SageMaker](https://blowstack.com/tools/cloud-services-hub/amazon-sagemaker)**: AWS Glue interactive sessions can be used from SageMaker Studio notebooks to develop and run serverless data preparation code. SageMaker's lakehouse capabilities also use the Glue Data Catalog to organize and access lakehouse tables.
- **[Amazon SageMaker Data Wrangler](https://blowstack.com/tools/cloud-services-hub/amazon-sagemaker-data-wrangler)**: AWS Glue Data Catalog databases and tables can be selected when importing data into Data Wrangler through Athena. This lets users prepare cataloged lake datasets for machine learning.
- **[Amazon Security Lake](https://blowstack.com/tools/cloud-services-hub/amazon-security-lake)**: AWS Glue Data Catalog stores the database and table definitions for Security Lake datasets. Security Lake maintains this metadata so authorized subscribers can discover and query normalized security logs.
- **[Amazon Simple Queue Service (Amazon SQS)](https://blowstack.com/tools/cloud-services-hub/amazon-simple-queue-service-(amazon-sqs))**: AWS Glue crawlers can consume S3 event notifications from an SQS queue to identify changed objects. Event-driven crawling reduces the need to list and inspect an entire S3 dataset on every crawl.
- **[Amazon Timestream](https://blowstack.com/tools/cloud-services-hub/amazon-timestream)**: AWS Glue jobs can use the Amazon Timestream connector to read time-series data and write transformed records. This supports ETL pipelines that load historical measurements into Timestream or process query results.
- **[Amazon VPC](https://blowstack.com/tools/cloud-services-hub/amazon-vpc)**: AWS Glue connections can specify VPC subnets and security groups for accessing private data sources. Glue creates network interfaces in the selected subnet so jobs and crawlers can reach resources such as private databases.
- **[AWS Application Discovery Service](https://blowstack.com/tools/cloud-services-hub/aws-application-discovery-service)**: AWS Glue crawlers and Data Catalog tables support exploration of Application Discovery Service data exported to Amazon S3. The discovery service's data exploration workflow uses these resources to make collected server data queryable.
- **[AWS Clean Rooms](https://blowstack.com/tools/cloud-services-hub/aws-clean-rooms)**: AWS Glue Data Catalog tables can be referenced when creating S3-backed configured tables in Clean Rooms. The catalog supplies the schema and data location used to associate a dataset with a collaboration.
- **[AWS CloudFormation](https://blowstack.com/tools/cloud-services-hub/aws-cloudformation)**: AWS Glue resources can be provisioned through CloudFormation, including databases, tables, jobs, crawlers, connections, triggers, and workflows. Templates can deploy a pipeline's catalog definitions and processing configuration together.
- **[AWS CloudTrail](https://blowstack.com/tools/cloud-services-hub/aws-cloudtrail)**: AWS Glue API activity is recorded by CloudTrail for auditing. These records help identify who changed catalog metadata, modified a job, or initiated a processing operation.
- **[AWS CodeCommit](https://blowstack.com/tools/cloud-services-hub/aws-codecommit)**: AWS Glue Studio supports synchronizing job scripts and associated job definitions with CodeCommit repositories. This provides version control for visual and script-based ETL development.
- **[AWS Config](https://blowstack.com/tools/cloud-services-hub/aws-config)**: AWS Config can record supported AWS Glue resource configurations and track changes over time. Configuration history and rules can help audit resources such as Glue jobs and security configurations.
- **[AWS Cost and Usage Report](https://blowstack.com/tools/cloud-services-hub/aws-cost-and-usage-report)**: AWS Glue crawlers and Data Catalog tables support the Athena integration for Cost and Usage Reports delivered to S3. Cataloging the report files makes billing line items available for SQL analysis.
- **[AWS Data Exchange](https://blowstack.com/tools/cloud-services-hub/aws-data-exchange)**: AWS Glue Data Catalog tables can be shared through AWS Data Exchange data products that use Lake Formation permissions. Subscribers receive governed access to the published tables without the provider distributing separate copies of the underlying data.
- **[AWS Database Migration Service (AWS DMS)](https://blowstack.com/tools/cloud-services-hub/aws-database-migration-service-(aws-dms))**: AWS Glue Data Catalog generation can be enabled for supported DMS S3 targets. DMS creates catalog metadata describing the migrated data so analytics tools can discover and query it.
- **[AWS Entity Resolution](https://blowstack.com/tools/cloud-services-hub/aws-entity-resolution)**: AWS Glue Data Catalog tables define S3-backed input datasets for Entity Resolution workflows. The table schemas provide the columns that are mapped to matching attributes.
- **[AWS Glue DataBrew](https://blowstack.com/tools/cloud-services-hub/aws-glue-databrew)**: AWS Glue Data Catalog tables can be used as DataBrew datasets for supported data sources. DataBrew recipe jobs can also publish supported output datasets into the catalog for subsequent ETL and analytics.
- **[AWS Identity and Access Management (IAM)](https://blowstack.com/tools/cloud-services-hub/aws-identity-and-access-management-(iam))**: AWS Glue uses IAM roles to access data sources, write outputs, and perform processing operations. IAM policies also control who can manage Glue jobs, crawlers, connections, and Data Catalog resources.
- **[AWS Key Management Service (AWS KMS)](https://blowstack.com/tools/cloud-services-hub/aws-key-management-service-(aws-kms))**: AWS Glue uses KMS keys to protect supported stored data, including Data Catalog metadata and connection passwords. Glue security configurations also specify encryption settings for job outputs, CloudWatch logs, and job bookmarks.
- **[AWS Marketplace](https://blowstack.com/tools/cloud-services-hub/aws-marketplace)**: AWS Glue can use data connectors distributed through AWS Marketplace. Subscribed connectors can be configured as Glue connections to access additional data sources and destinations.
- **[AWS PrivateLink](https://blowstack.com/tools/cloud-services-hub/aws-privatelink)**: AWS Glue supports interface VPC endpoints powered by PrivateLink. Clients in a VPC can call supported Glue APIs without sending traffic over the public internet.
- **[AWS Resource Access Manager (AWS RAM)](https://blowstack.com/tools/cloud-services-hub/aws-resource-access-manager-(aws-ram))**: AWS Glue Data Catalog resources can be shared across accounts through Lake Formation integrations that use AWS RAM resource shares. Recipient accounts can access authorized databases and tables without duplicating their metadata.
- **[AWS Secrets Manager](https://blowstack.com/tools/cloud-services-hub/aws-secrets-manager)**: AWS Glue connections can reference Secrets Manager secrets for database and connector credentials. Jobs and crawlers retrieve the credentials using their execution roles rather than embedding passwords in scripts.
- **[AWS Security Token Service (AWS STS)](https://blowstack.com/tools/cloud-services-hub/aws-security-token-service-(aws-sts))**: AWS Glue assumes its configured IAM execution roles through STS to obtain temporary credentials. Those credentials authorize jobs and crawlers to access their permitted data sources and destinations.
- **[AWS Step Functions](https://blowstack.com/tools/cloud-services-hub/aws-step-functions)**: AWS Glue jobs can be started through the Step Functions service integration, including a synchronous mode that waits for a job run to finish. State machines can coordinate ETL stages and handle job failures with retries or recovery branches.

## What is AWS Glue

AWS Glue is a cloud-based service provided by Amazon Web Services (AWS) that serves as a fully managed extract, transform, and load (ETL) solution. It is designed to streamline the process of preparing and combining data for analytics and data processing workflows.

AWS Glue automates the time-consuming steps of data preparation, such as discovering, categorizing, cleaning, enriching, and moving data between various data stores. Its Data Catalog is a centralized technical metadata repository, with AWS Lake Formation capabilities for centralized data governance. This catalog provides a unified view of all data sources and simplifies the management of data across different AWS services and your data warehouse.

AWS Glue Data Quality can recommend rules for data lakes, evaluate rules in pipelines, and calculate quality scores visible in the Data Catalog. Rules can be customized and scheduled, with actions to alert teams when quality issues occur.

AWS Glue dynamically generates ETL code in Python or Scala, which can be further customized and stored for future use. It is serverless, meaning that it automatically provisions the compute resources required to process data according to the workload. This makes it a scalable solution that adjusts to the volume of data, whether it’s processing a few records or terabytes of data. Glue is integrated with a wide range of AWS services, making it a versatile tool for different ETL tasks and enabling seamless data integration across the AWS ecosystem.

AWS Glue has standard and flexible execution classes. Standard execution is intended for time-sensitive workloads requiring fast startup and dedicated resources; Flex suits nonurgent jobs whose start and completion times may vary.

For interactive development, Glue Interactive Sessions let you explore, experiment on, and process data using your preferred IDE or notebook.

Custom readers, writers, and transformations can be imported into Glue ETL jobs as libraries.

Glue Zero-ETL defaults to 25 integrations per account, with up to 15 integrations associated with each source and up to 50 associated with each target.

In addition to ETL capabilities, AWS Glue offers features like the Glue DataBrew, which allows users to clean and normalize data without writing code. It provides a visual interface to easily combine, transform, and categorize data. AWS Glue's ability to handle both batch and streaming data makes it suitable for various data processing needs, from big data analytics to real-time analytics applications.

For streaming sources, Data Catalog tables and data stream properties are defined manually rather than populated by a data-store crawler.

This comprehensive service simplifies data preparation and integration, facilitating more efficient and effective data analysis and decision-making processes within organizations.

## Free AWS Glue tools

- [SQL Date Format Converter](https://blowstack.com/tools/aws-sql-date-format-converter): Get the expression that reads a date or timestamp in Athena, Redshift and Glue, and fix date errors.

## Concepts related to AWS Glue

- [ACID Transactions](https://blowstack.com/tools/cloud-concepts-hub/acid-transactions): ACID transactions group data operations into a reliable unit of work with atomicity, consistency, isolation, and durability guarantees.
- [Algorithmic Complexity](https://blowstack.com/tools/cloud-concepts-hub/algorithmic-complexity): Algorithmic complexity describes how an algorithm’s resource requirements grow as the size of its input increases.
- [Analytics](https://blowstack.com/tools/cloud-concepts-hub/analytics): Analytics is the systematic examination of data to answer questions, identify patterns, and support decisions.
- [Apache Parquet](https://blowstack.com/tools/cloud-concepts-hub/apache-parquet): Apache Parquet is an open, column-oriented binary file format designed for efficient storage and analytical processing of structured and nested data.
- [API Versioning](https://blowstack.com/tools/cloud-concepts-hub/api-versioning): API versioning is the practice of identifying and managing distinct API contracts so interfaces can evolve without unexpectedly breaking consumers.
- [Backward Compatibility](https://blowstack.com/tools/cloud-concepts-hub/backward-compatibility): Backward compatibility is the ability of an updated system, interface, or data reader to continue supporting the contracts or data of earlier versions.
- [Batch Processing](https://blowstack.com/tools/cloud-concepts-hub/batch-processing): Batch processing collects data into bounded groups and processes each group as a job rather than handling every record immediately as it arrives.
- [Batching](https://blowstack.com/tools/cloud-concepts-hub/batching): Batching groups multiple records, requests, or work items into a unit for more efficient transfer, storage, or processing.
- [Binning](https://blowstack.com/tools/cloud-concepts-hub/binning): Binning converts numerical values into discrete intervals so machine learning models can use ranges rather than exact measurements.
- [Broadcast Joins](https://blowstack.com/tools/cloud-concepts-hub/broadcast-joins): Broadcast joins replicate a small join input to participating workers so each worker can join it locally with its portion of a larger input.
- [Capability Maturity Model](https://blowstack.com/tools/cloud-concepts-hub/capability-maturity-model): A Capability Maturity Model is a framework for assessing how consistently an organization performs a capability and identifying the improvements needed to make it repeatable, measurable, and continuously optimized.
- [Change Readiness](https://blowstack.com/tools/cloud-concepts-hub/change-readiness): Change Readiness is an organization’s willingness and ability to adopt and sustain a new way of working, including the people, processes, data, technology, and governance needed for AI adoption.
- [Checkpointing](https://blowstack.com/tools/cloud-concepts-hub/checkpointing): Checkpointing is the practice of durably saving processing progress and any required execution state so work can resume from a known recovery point.
- [Class Imbalance](https://blowstack.com/tools/cloud-concepts-hub/class-imbalance): Class imbalance occurs when some target classes in a dataset have substantially more examples than others.
- [Class Resampling](https://blowstack.com/tools/cloud-concepts-hub/class-resampling): Class resampling changes how frequently examples from each target class appear during training to reduce the effects of class imbalance.
- [Columnar Storage](https://blowstack.com/tools/cloud-concepts-hub/columnar-storage): Columnar storage organizes data by column rather than by complete row, allowing analytical workloads to read and compress selected fields efficiently.
- [Comma-Separated Values](https://blowstack.com/tools/cloud-concepts-hub/comma-separated-values): Comma-Separated Values (CSV) is a plain-text format that represents tabular records as fields separated by commas.
- [Cron Expressions](https://blowstack.com/tools/cloud-concepts-hub/cron-expressions): Cron expressions are compact, field-based patterns that define recurring execution times on a calendar.
- [Data Anonymization](https://blowstack.com/tools/cloud-concepts-hub/data-anonymization): Data anonymization transforms information so that individuals are no longer reasonably identifiable while preserving useful characteristics for analysis and AI.
- [Data Augmentation](https://blowstack.com/tools/cloud-concepts-hub/data-augmentation): Data augmentation creates additional training examples by transforming existing data or generating controlled variations that preserve task-relevant meaning and valid labels.
- [Data Cataloging](https://blowstack.com/tools/cloud-concepts-hub/data-cataloging): Data cataloging is the practice of maintaining a searchable inventory of data assets and metadata describing their meaning, ownership, origins, and permitted use.
- [Data Compression](https://blowstack.com/tools/cloud-concepts-hub/data-compression): Data compression represents information using fewer bytes to reduce storage consumption and data movement.
- [Data Contracts](https://blowstack.com/tools/cloud-concepts-hub/data-contracts): Data contracts are explicit agreements between data producers and consumers that define a dataset’s structure, meaning, quality expectations, usage conditions, and change-management rules.
- [Data Curation](https://blowstack.com/tools/cloud-concepts-hub/data-curation): Data curation is the deliberate selection, validation, organization, and ongoing maintenance of data so it is trustworthy and suitable for an intended use.
- [Data Deduplication](https://blowstack.com/tools/cloud-concepts-hub/data-deduplication): Data deduplication identifies repeated representations of the same information and removes or consolidates them according to defined matching and retention rules.
- [Data governance](https://blowstack.com/tools/cloud-concepts-hub/data-governance): Data governance is the system of policies, responsibilities, and controls that determines how data is collected, used, protected, and managed throughout its lifecycle.
- [Data Integrity](https://blowstack.com/tools/cloud-concepts-hub/data-integrity): Data integrity is the assurance that data remains accurate, complete, consistent, and protected against unauthorized or unintended changes throughout its lifecycle.
- [Data Lakes](https://blowstack.com/tools/cloud-concepts-hub/data-lakes): A data lake is a shared repository that stores large volumes of structured, semi-structured, and unstructured data for analytics and other processing.
- [Data Mesh](https://blowstack.com/tools/cloud-concepts-hub/data-mesh): Data Mesh is a decentralized data operating model in which business domains own data products while sharing a self-service platform and federated governance standards.
- [Data Network Effects](https://blowstack.com/tools/cloud-concepts-hub/data-network-effects): Data network effects occur when additional usage generates data that improves a product or service, increasing its value and encouraging further usage.
- [Data Partitioning](https://blowstack.com/tools/cloud-concepts-hub/data-partitioning): Data partitioning divides a dataset into smaller subsets using defined rules so that storage, ingestion, processing, and queries can operate on manageable portions of the data.
- [Data Profiling](https://blowstack.com/tools/cloud-concepts-hub/data-profiling): Data profiling is the systematic examination of a dataset’s structure, values, distributions, and relationships to understand its characteristics and identify potential quality problems.
- [Data Provenance](https://blowstack.com/tools/cloud-concepts-hub/data-provenance): Data provenance is the documented origin, ownership, and history of data, including how it was collected, transformed, and used.
- [Data Quality](https://blowstack.com/tools/cloud-concepts-hub/data-quality): Data quality is the degree to which data is accurate, complete, consistent, valid, timely, and suitable for its intended use.
- [Data Representativeness](https://blowstack.com/tools/cloud-concepts-hub/data-representativeness): Data representativeness is the degree to which a dataset reflects the people, conditions, and patterns relevant to an AI system’s intended use.
- [Data Shuffling](https://blowstack.com/tools/cloud-concepts-hub/data-shuffling): Data shuffling is the random reordering of records to reduce unintended ordering effects during dataset preparation and machine learning training.
- [Data Silos](https://blowstack.com/tools/cloud-concepts-hub/data-silos): Data silos are isolated collections of data that are difficult for other teams or systems to discover, access, interpret, or combine.
- [Data Skew](https://blowstack.com/tools/cloud-concepts-hub/data-skew): Data skew is an uneven distribution of data or data-processing work across partitions, tasks, or nodes in a distributed system.
- [Data Validation](https://blowstack.com/tools/cloud-concepts-hub/data-validation): Data validation is the process of checking data against defined structural, quality, and business rules before accepting or using it.
- [Deduplication](https://blowstack.com/tools/cloud-concepts-hub/deduplication): Deduplication identifies repeated representations of the same data and removes or consolidates them according to explicit matching and retention rules.
- [Denormalization](https://blowstack.com/tools/cloud-concepts-hub/denormalization): Denormalization deliberately combines related data or stores redundant values to make common reads require less work.
- [Directed Acyclic Graphs](https://blowstack.com/tools/cloud-concepts-hub/directed-acyclic-graphs): Directed acyclic graphs model tasks and their dependencies using one-way connections that never form a cycle.
- [Distributed Processing](https://blowstack.com/tools/cloud-concepts-hub/distributed-processing): Distributed processing divides computational work across multiple networked machines that coordinate to produce a result.
- [Event-Time Processing](https://blowstack.com/tools/cloud-concepts-hub/event-time-processing): Event-time processing transforms and aggregates data according to when events occurred rather than when a processing system receives or handles them.
- [Exception Handling](https://blowstack.com/tools/cloud-concepts-hub/exception-handling): Exception handling is the controlled detection and management of runtime failures so a program can recover, release resources, or terminate with useful diagnostic information.
- [Executive Sponsorship](https://blowstack.com/tools/cloud-concepts-hub/executive-sponsorship): Executive sponsorship is active, accountable support from a senior leader who authorizes resources, aligns stakeholders, and removes organizational barriers to a strategic initiative.
- [Extract, Transform, Load](https://blowstack.com/tools/cloud-concepts-hub/extract-transform-load): Extract, Transform, Load is a data integration process that retrieves data from source systems, prepares it for a target use case, and writes it to a destination.
- [Feature Engineering](https://blowstack.com/tools/cloud-concepts-hub/feature-engineering): Feature engineering is the process of creating, transforming, and selecting input variables so machine learning models can learn useful patterns from data.
- [Gap Analysis](https://blowstack.com/tools/cloud-concepts-hub/gap-analysis): Gap analysis compares current capabilities with a defined target state to identify what must change to achieve an intended outcome.
- [Generators](https://blowstack.com/tools/cloud-concepts-hub/generators): Generators are resumable routines that produce a sequence of values on demand instead of constructing the entire sequence in memory.
- [Incremental Loading](https://blowstack.com/tools/cloud-concepts-hub/incremental-loading): Incremental loading ingests only new or changed data since a previously recorded processing point instead of reloading an entire dataset.
- [Incremental Processing](https://blowstack.com/tools/cloud-concepts-hub/incremental-processing): Incremental processing updates derived data by processing only new or changed inputs and their affected results instead of recomputing the entire dataset.
- [Incremental Transfer](https://blowstack.com/tools/cloud-concepts-hub/incremental-transfer): Incremental transfer moves only new or changed data after an established baseline instead of repeatedly transferring the entire dataset.
- [Integration Testing](https://blowstack.com/tools/cloud-concepts-hub/integration-testing): Integration testing verifies that connected software components exchange data and work together correctly across their interfaces.
- [JDBC](https://blowstack.com/tools/cloud-concepts-hub/jdbc): JDBC (Java Database Connectivity) is a standard Java API for connecting to databases, executing SQL, and processing results through database-specific drivers.
- [Joins](https://blowstack.com/tools/cloud-concepts-hub/joins): Joins combine records from two or more datasets using a condition that defines how the records relate.
- [JSON](https://blowstack.com/tools/cloud-concepts-hub/json): JSON is a text-based data interchange format that represents structured values using objects, arrays, strings, numbers, booleans, and null.
- [Key Salting](https://blowstack.com/tools/cloud-concepts-hub/key-salting): Key salting adds a controlled extra value to a key so that records sharing a hot key can be distributed across multiple processing or storage partitions.
- [Latency](https://blowstack.com/tools/cloud-concepts-hub/latency): Latency is the elapsed time between a defined starting event and the corresponding response, completion, or availability of a result.
- [Leading Indicators](https://blowstack.com/tools/cloud-concepts-hub/leading-indicators): Leading indicators are measurable signals that precede an outcome and help predict whether an AI initiative will achieve its intended business value.
- [Lemmatization](https://blowstack.com/tools/cloud-concepts-hub/lemmatization): Lemmatization maps inflected word forms to their dictionary form, or lemma, using linguistic rules, vocabulary, and sometimes grammatical context.
- [Log Normalization](https://blowstack.com/tools/cloud-concepts-hub/log-normalization): Log normalization converts differently structured log records into a consistent schema and representation so they can be searched, compared, and correlated.
- [Min-Max Scaling](https://blowstack.com/tools/cloud-concepts-hub/min-max-scaling): Min-Max Scaling is a feature transformation that uses minimum and maximum reference values to linearly map numeric data to a chosen range, commonly 0 to 1.
- [Missing Value Imputation](https://blowstack.com/tools/cloud-concepts-hub/missing-value-imputation): Missing value imputation replaces absent data values with justified estimates or explicit placeholders so datasets can be analyzed or used by machine learning systems.
- [Null Handling](https://blowstack.com/tools/cloud-concepts-hub/null-handling): Null handling is the explicit detection, interpretation, and treatment of missing or unknown values during data processing.
- [Open Cybersecurity Schema Framework (OCSF)](https://blowstack.com/tools/cloud-concepts-hub/open-cybersecurity-schema-framework-(ocsf)): Open Cybersecurity Schema Framework (OCSF) is an open, vendor-neutral framework for representing cybersecurity events in a common schema.
- [Open Database Connectivity (ODBC)](https://blowstack.com/tools/cloud-concepts-hub/open-database-connectivity-(odbc)): Open Database Connectivity (ODBC) is a standard programming interface that lets applications access data sources through compatible drivers.
- [Optimistic Concurrency Control](https://blowstack.com/tools/cloud-concepts-hub/optimistic-concurrency-control): Optimistic Concurrency Control lets concurrent operations proceed independently and validates their assumptions before committing changes.
- [ORC File Format](https://blowstack.com/tools/cloud-concepts-hub/orc-file-format): ORC File Format is an open, columnar binary format that stores typed data with compression and statistics for efficient analytical processing.
- [Parallel Processing](https://blowstack.com/tools/cloud-concepts-hub/parallel-processing): Parallel processing divides work into tasks that execute simultaneously across multiple processing resources to reduce completion time or increase throughput.
- [Partition Pruning](https://blowstack.com/tools/cloud-concepts-hub/partition-pruning): Partition pruning is the exclusion of data partitions that cannot satisfy a query’s filters, reducing the amount of data scanned.
- [Partitioning](https://blowstack.com/tools/cloud-concepts-hub/partitioning): Partitioning divides data or a stream of records into smaller, addressable groups so storage, ingestion, and processing can scale efficiently.
- [Point-in-Time Joins](https://blowstack.com/tools/cloud-concepts-hub/point-in-time-joins): Point-in-Time Joins combine records with the feature values that were valid and available at each record’s historical prediction time.
- [Predicate Pushdown](https://blowstack.com/tools/cloud-concepts-hub/predicate-pushdown): Predicate pushdown is a query optimization that moves filtering closer to the data source so fewer records reach later processing stages.
- [Pseudonymization](https://blowstack.com/tools/cloud-concepts-hub/pseudonymization): Pseudonymization replaces identifying data with substitute values so records cannot be attributed to a person without separately protected additional information.
- [Random Sampling](https://blowstack.com/tools/cloud-concepts-hub/random-sampling): Random sampling selects observations from a defined population using chance rather than their position, convenience, or an engineer’s judgment.
- [Referential Integrity](https://blowstack.com/tools/cloud-concepts-hub/referential-integrity): Referential integrity ensures that references between related records point to existing, valid target records.
- [Relational Joins](https://blowstack.com/tools/cloud-concepts-hub/relational-joins): Relational joins combine rows from two datasets according to a matching condition, usually involving shared keys.
- [Repartitioning](https://blowstack.com/tools/cloud-concepts-hub/repartitioning): Repartitioning redistributes data among processing or storage partitions to change parallelism, balance workloads, or align data with downstream operations.
- [Schema Evolution](https://blowstack.com/tools/cloud-concepts-hub/schema-evolution): Schema evolution is the controlled modification of data structures over time while managing compatibility between existing data, producers, and consumers.
- [Schema on Read](https://blowstack.com/tools/cloud-concepts-hub/schema-on-read): Schema on read is an approach that applies a schema when data is read or queried rather than requiring the storage layer to enforce that schema when data is written.
- [Schema Validation](https://blowstack.com/tools/cloud-concepts-hub/schema-validation): Schema validation checks whether data conforms to an explicitly defined structure, including required fields, data types, and permitted values.
- [Semi-Structured Data](https://blowstack.com/tools/cloud-concepts-hub/semi-structured-data): Semi-structured data uses keys, tags, or other markers to organize information without requiring every record to follow the same fixed tabular schema.
- [Serialization](https://blowstack.com/tools/cloud-concepts-hub/serialization): Serialization converts in-memory data into a defined text or binary representation for storage, transmission, and later reconstruction.
- [Slowly Changing Dimensions](https://blowstack.com/tools/cloud-concepts-hub/slowly-changing-dimensions): Slowly Changing Dimensions is a data-modeling approach for deciding how changes to descriptive attributes are overwritten, retained, or versioned over time.
- [Source Citation](https://blowstack.com/tools/cloud-concepts-hub/source-citation): Source citation is the practice of identifying and documenting the origins of information so that data, model artifacts, and AI-generated claims can be traced to supporting sources.
- [SQL Joins](https://blowstack.com/tools/cloud-concepts-hub/sql-joins): SQL joins combine rows from tables or other relational inputs according to a matching condition and rules for retaining unmatched rows.
- [Star Schema](https://blowstack.com/tools/cloud-concepts-hub/star-schema): A star schema is an analytical data model that connects a central fact table to descriptive dimension tables through keys.
- [Stateful Processing](https://blowstack.com/tools/cloud-concepts-hub/stateful-processing): Stateful processing retains information across records, requests, or events so that earlier activity can influence later results.
- [Stratified Sampling](https://blowstack.com/tools/cloud-concepts-hub/stratified-sampling): Stratified sampling divides a population into defined, nonoverlapping groups and randomly selects records from each group to create a sample.
- [Structured Data](https://blowstack.com/tools/cloud-concepts-hub/structured-data): Structured data is information organized according to a defined schema, with consistent fields, data types, and relationships that support systematic processing.
- [Structured Logging](https://blowstack.com/tools/cloud-concepts-hub/structured-logging): Structured logging records application events as consistently named, machine-readable fields rather than relying only on free-form text messages.
- [Throughput](https://blowstack.com/tools/cloud-concepts-hub/throughput): Throughput is the amount of data or work a system successfully processes per unit of time.
- [Train-Validation-Test Split](https://blowstack.com/tools/cloud-concepts-hub/train-validation-test-split): A train-validation-test split divides a dataset into separate subsets for fitting a model, selecting its configuration, and evaluating its performance on unseen data.
- [Training-Serving Skew](https://blowstack.com/tools/cloud-concepts-hub/training-serving-skew): Training-serving skew is a mismatch between the data or feature processing used to train a model and what the model receives during inference.
- [Tree Data Structures](https://blowstack.com/tools/cloud-concepts-hub/tree-data-structures): Tree data structures organize nodes into an acyclic hierarchy of parent–child relationships for efficient navigation, searching, and transformation.
- [Uniqueness Constraints](https://blowstack.com/tools/cloud-concepts-hub/uniqueness-constraints): Uniqueness constraints are rules that require a column value or combination of column values to occur no more than once within a defined scope.
- [Vectorization](https://blowstack.com/tools/cloud-concepts-hub/vectorization): Vectorization expresses operations over arrays or batches of values so optimized execution engines can process them with less per-element overhead.
- [Window Functions](https://blowstack.com/tools/cloud-concepts-hub/window-functions): Window functions calculate values across related rows while preserving each row in the query result.
- [Windowed Aggregation](https://blowstack.com/tools/cloud-concepts-hub/windowed-aggregation): Windowed aggregation computes summaries over bounded groups of events, typically defined by time, so continuously arriving data can become usable metrics and machine-learning features.
