AWS Certified Data Engineer - Associate (DEA-C01) Course

Amazon Kinesis Data Streams accepts records from many producers into a retained stream. This is fan-in: producers contribute records to the same stream rather than sending a separate copy directly to every consumer.

Amazon MSK runs Apache Kafka clusters for applications. A producer publishes records to a named topic, such as orders. Several producers can publish to the same topic: this is fan-in. It brings their records into one named stream, but does not automatically join or aggregate their contents.

Both Amazon Kinesis Data Streams and Amazon MSK can combine events from multiple producers and let several applications read the retained events. Reading does not remove a record. An alerting application and an analytics application can therefore process the same data at different speeds.

An event source mapping is a Lambda resource that connects a Kinesis stream to a function. With a standard stream consumer, Lambda polls the stream's shards, gathers records into batches, and invokes the function synchronously. Kinesis does not directly invoke the function for each arriving record.

Amazon Kinesis Data Streams retains records for consumers to read. A Lambda function can process those records through a Lambda event source mapping, which reads the stream and invokes the function with batches. Processing does not remove the source records; they remain available within the retention period.

A database migration must move existing rows without losing writes that happen during the move. The full load creates the initial baseline on the target. Change Data Capture (CDC) captures inserts, updates, and deletes so the target can catch up while applications continue writing to the source.

DynamoDB Streams captures item-level changes after you enable it on a table. Records identify changes as INSERT, MODIFY, or REMOVE. An update that leaves the item's data unchanged does not produce a stream record. Enabling a stream does not produce an initial snapshot of existing items.

AWS Database Migration Service (AWS DMS) moves database data from a source to a target. In this example, a replication task runs on a provisioned replication instance and delivers MySQL order data to an Amazon S3 landing bucket for downstream processing.

The AWS Glue Data Catalog is a shared technical metadata store. A database groups table definitions; it is not a container of copied source records. Engines such as Athena, EMR, and Glue jobs can reference these definitions instead of each maintaining a separate description of the same data.

An AWS Glue connection is a named, reusable definition of how to reach a data store. It stores connection properties, authentication settings, and, when needed, VPC settings. It is not a running network tunnel and does not copy data. A crawler or ETL job uses the definition when it connects.

An AWS Glue batch ETL job reads a bounded set of source data, transforms its records, and writes results to a target. A DynamicFrame is Glue's distributed collection of records for this work; it is not a storage location.

For this lesson, consider Amazon EMR on EC2. A batch job can read objects directly from Amazon S3 through its framework's S3 connector. It does not have to copy the entire input into HDFS first.

COPY loads data from S3 objects into an existing Redshift table. UNLOAD runs a query and writes its results to S3 objects. These commands move data in bulk rather than sending individual rows through a client application.

Amazon Redshift Spectrum lets Redshift SQL queries read data stored in Amazon S3 without first loading that data into warehouse tables. An external table describes the files; it does not contain a warehouse copy of their rows.

Amazon AppFlow uses connectors to transfer data between supported applications and AWS destinations without you writing the API integration. A connection holds the authentication configuration for an application; a flow uses that connection and defines what data to move, where to send it, and when to run.

An S3 event notification connects selected bucket events to a destination. For a direct Lambda trigger, configure an event type, optional object-key filters, and a Lambda function in the same AWS Region as the bucket. S3 invokes the function asynchronously with event metadata, not the object's contents.

A Lambda handler is the entry point that receives an event and runs your processing code. With a direct S3 notification, S3 invokes Lambda asynchronously: Lambda accepts the notification and queues it internally before running the handler. The event identifies the bucket and object key; it does not contain the object's contents. Your code reads the object when needed.

A batch is a bounded set of inputs with a defined completion point. Define its membership before processing: an explicit file list called an input manifest, a consistent snapshot, or an extraction range. For time ranges, use one shared boundary convention, such as timestamp >= start AND timestamp < end, so adjacent ranges do not overlap.

An AWS Glue Spark job runs your transformation script using compute that Glue provisions and manages. You choose the runtime and worker settings rather than creating and maintaining a Spark cluster yourself.

Incremental loading processes changes since saved progress, rather than the whole source. After an initial baseline load, a pipeline can use a watermark as its last successfully processed boundary. For each run, retain that lower boundary, L, and capture a fixed upper boundary, U. Read changes satisfying L < change_position <= U; changes beyond U belong to a later batch.

A predicate is a condition that decides which rows qualify, such as status = 'complete'. Predicate pushdown lets an execution engine pass a supported condition to a database or file scan instead of bringing all rows into a transformation job and filtering afterward. Fewer rows reaching the job can mean less network traffic, decoding, memory use, and downstream processing.

A batch ingestion job has three different kinds of partitioning. Source-read partitions divide the input among readers. Execution partitions divide work among processing tasks. Persisted output partitions organize the stored result. Their counts do not have to match, and changing one does not automatically change the others.

REST, or Representational State Transfer, is an architectural style, not a protocol or a product. A resource is something an application exposes, such as an order. A URI such as https://api.example.com/orders/123 identifies it. A representation describes its state—for example, a JSON document containing its identifier, items, and status. JSON is common, but REST does not require it.

Pagination divides a result collection into pages. A single API request usually returns one page, not the whole collection. To traverse the results, request the first page, process its records, and use the response's documented continuation information to request the next page.

A reliable API consumer follows the provider's request contract: the endpoint, HTTP method, parameters, authentication scheme, and response format. Use HTTPS and grant the calling identity only the read permissions it needs. Authentication establishes the caller's identity; authorization determines which data that identity may read.

Set an API Gateway method or route to AWS_IAM when callers should use AWS identities. The client signs its HTTP request using Signature Version 4 and AWS credentials. Temporary credentials also require a session token. A signing-capable SDK or HTTP client can construct the signature; the secret access key is not sent with the request.

A directed acyclic graph, or DAG, represents tasks as vertices and dependencies as directed edges. Writing O → V means O is upstream of V: V must satisfy its dependency on O before becoming eligible to run. The edge describes a dependency, not necessarily a transfer of data.

A cron expression selects recurring calendar times; it does not describe the work to run. Start by writing the requirement in plain words, including its time zone: for example, “02:00 Monday through Friday, UTC.” Cron does not itself define dependencies, retries, or successful completion.

Amazon EventBridge Scheduler invokes a configured target API according to a schedule. It does not need an incoming event or an event-bus rule. For data processing, it can request that a job or crawler start; it does not itself perform the processing.

An AWS Glue workflow groups related jobs, crawlers, and triggers into a coordinated process. Jobs perform the transformations; crawlers discover metadata. Triggers decide when those activities start, and the workflow gives you a graph and run history for the process.

In Amazon MWAA, Apache Airflow orchestrates work defined in a Python DAG: a directed acyclic graph of tasks and their dependencies. The DAG is a reusable definition; a DAG run is one execution of that definition. Each task has a separate task instance within each run.

Event-triggered ingestion starts processing when a source reports a change, rather than waiting for a scheduled scan. For Amazon S3, an object-created event identifies an arriving object. The event contains metadata, not the object's file contents; an ingestion consumer uses that information to retrieve the object when needed.

An event bus receives events describing things that happened. A rule uses an event pattern to select events and sends matches to its configured targets. Producers do not need to call those targets themselves.

An event source mapping is a Lambda resource that connects a Kinesis data stream to a function. Lambda manages reading the stream and invoking your code; your function does not need its own polling loop.

At-least-once delivery allows the same record to reach a consumer more than once during recovery from failures. For Kinesis-to-Lambda processing, the important boundary is between records accepted into Kinesis Data Streams and invocation of the consumer function—not successful completion of every business operation.

Idempotency means that processing the same logical event again does not add another intended effect. A worker can run twice while creating only one accepted-event row. This matters when a write succeeds but its response is lost: the worker cannot safely assume that failure to receive a response means nothing happened.

Classless Inter-Domain Routing (CIDR) describes an address range using an address and a prefix length, such as 203.0.113.0/24. IPv4 addresses have 32 bits. The /24 prefix fixes the first 24 bits and leaves 8 bits free to vary.

A data source can use an IP allowlist to accept connections only from approved source addresses. The important address is the source IP arriving at that data source, not necessarily the address configured on the worker making the connection.

An IP allowlist permits connections from specified source addresses; other sources are not permitted by that rule set. For a data-source connection, identify the machine initiating the connection and the address seen where the destination enforces its rule. The database's own address is the destination, not the source to allow.

A security group controls traffic for associated network interfaces, including those used by EC2 instances and RDS databases. Its rules allow traffic; they do not explicitly deny it. For a new connection, the client's outbound rules and the data source's inbound rules must both allow the traffic.

Exponential backoff increases the waiting time between retries of a failed operation. Before choosing a delay, decide whether the failure is retryable. Throttling, selected server errors, and connection failures can be temporary. Invalid parameters and permission errors generally need correction, not another identical request. Follow the API's documented error classifications and server retry guidance.

An ingestion producer must budget the work it sends, not just react when a service rejects it. A sustained rate describes work allowed over time. A burst allowance permits a short spike above that rate when capacity or credit is available. A quiet period does not create unlimited burst credit, and not every service guarantees burst capacity.

Exponential backoff answers: “After this attempt fails, how long should the client wait before trying again?” It is reactive: a retryable failure triggers a delay, and repeated failures lead to progressively larger delay intervals, usually up to a configured maximum. It does not, by itself, govern first attempts.

Backpressure makes upstream work respond to downstream capacity. When a destination cannot keep up, the component feeding it reduces concurrency, delays requests, or pauses reading instead of creating unlimited work.

Rate limiting enforces a request-rate policy: how much traffic a caller or workload is allowed to send over time. Excess requests may be rejected or delayed. The policy can protect capacity or allocate a fair share, but a configured limit does not necessarily track what a downstream system can currently process.

Throughput is the work DynamoDB can serve over time. Capacity consumption depends on more than request count: larger items consume more capacity, and read consistency and transactional operations also affect consumption. Read and write demand must be considered separately. Throttling means DynamoDB cannot serve some requested work within an applicable capacity limit; the client must handle that work rather than assume it succeeded.

In publish-subscribe messaging, a publisher sends an event to a topic rather than contacting every recipient. A subscription connects that topic to a destination. Amazon Simple Notification Service, or Amazon SNS, provides topic-based messaging on AWS.

A work queue divides work among competing consumers. Several workers read from the same queue, but each message is intended to be processed by one worker, not by every worker. Amazon SQS provides this queued-work pattern.

Amazon SNS distributes notifications using a topic. A publisher sends a message to the topic, and SNS pushes it to subscribed endpoints. Choose this model when one event should notify several independent recipients, such as an email subscriber and an application.

Fan-in brings multiple producers into one logical destination. That destination can still use many shards and workers: shared does not mean single-threaded. Its capacity must accommodate the producers' combined traffic, not just the busiest producer.

Fan-out gives each interested application its own copy of an event. If 100 events arrive each second and two branches both receive every event, each branch receives 100 events per second—not 50. Together they must handle 200 initial deliveries per second, before retries.

A checkpoint records enough processing progress to restart from a known boundary. For a partitioned stream, that usually includes positions for each partition. A stateful calculation also needs saved state, such as running totals, that agrees with those positions.

ACID transactions protect a logical unit of database work. Idempotency means repeating the same logical request does not change its intended effect beyond the first successful application. One asks whether related changes are correct together; the other asks what happens when work is repeated.

A stateless worker can handle an ingestion request without remembering an earlier invocation. Its input must identify both the work and the data needed to perform it. For example, a message can carry a request ID, an S3 bucket and object key, an object version ID, and a transformation version.

State is information kept from earlier inputs so the next input can use it. In per-key state, each customer, device, or order has its own retained context. Choose a key that matches the entity whose history matters: a customer ID for customer totals, for example.

Stateful processing handles a new input using context retained from earlier inputs. A running purchase total needs the previous total; recognizing a repeated event needs a record of identifiers already seen. That retained information is state. Without it, the operation cannot produce the required result correctly.

A transaction is a logical unit of work that commits or aborts. During ingestion, one batch might insert event rows and update a batch-summary row. If those changes must agree, put them in the same transaction boundary: begin the transaction, perform both updates, and commit only when the whole unit succeeds.

Amazon Managed Service for Apache Flink runs a streaming application as a graph of operators. An operator performs a processing step; its parallel subtasks divide the work across runtime workers. This is distributed execution, not one worker processing the entire stream.

A stateless computation needs only its current input: converting a measurement from Celsius to Fahrenheit does not require earlier measurements. A stateful computation retains information across inputs, such as a running count or the measurements contributing to a time-window average.

Container CPU settings influence both where work runs and how much processor time it can receive. In Kubernetes on Amazon EKS, a CPU request is used for scheduling and influences CPU sharing under contention. A CPU limit caps CPU time. A container can use more than its request when capacity is available, up to its limit if one is set.

Right-sizing a container means choosing CPU and memory allocations that meet its performance needs without unnecessary capacity. Start with measurements from a representative workload: normal operation, busy periods, startup, and demanding jobs. For a data-processing container, include realistic input sizes and concurrent jobs, not just a small sample file.

Horizontal scaling changes how many workers serve a workload. Adding workers is scaling out; removing them is scaling in. Each worker contributes capacity by handling a different piece of work at the same time as other workers.

Right-sizing asks whether resource capacity fits the work. For containers, this includes the CPU and memory configured for each replica. A suitable configuration meets performance requirements with justified headroom; it is not necessarily the smallest configuration. Right-sizing can increase, decrease, or rebalance resources.

An Amazon ECS task definition describes the containers to run and their resource settings. A task is a running instance of that definition. For data processing, task sizing affects how quickly each worker can transform, compress, or aggregate records.

In Amazon EKS, a pod runs one or more containers on a worker node. For CPU and memory, each container can declare a request and a limit. The Kubernetes scheduler uses requests to find a node with enough unallocated capacity. It does not place pods based on their current low usage. Requests that are too large waste scheduling capacity; requests that are too small encourage crowding and resource contention.

AWS Fargate provides compute for an ECS task or an EKS pod using the resources you configure. Containers in that task or pod share its available resources, so include supporting containers, such as logging sidecars, when estimating demand.

Running containers involves two different jobs. Orchestration coordinates which containers run and manages their lifecycle. Compute supplies the resources that execute them. Amazon ECS is a managed container orchestrator; AWS Fargate is managed container compute.

A container orchestrator turns a workload description into running containers and manages their lifecycle. Amazon ECS uses AWS-native APIs and workload objects. Amazon EKS provides managed Kubernetes, so you describe work using the Kubernetes API. Both can run containerized data processors; the main distinction is how you configure and operate them.

JDBC and ODBC both let software connect to data sources, execute statements, and read results. They define application programming interfaces, or APIs: the operations an application can call. Neither is a database or a universal database network protocol.

JDBC is a Java API for opening connections, running SQL, and reading results. A JDBC driver implements that API for a database or query engine. Applications can use it directly; data-processing tools such as Spark use it underneath their readers and writers, including when the job code is Python.

Open Database Connectivity (ODBC) is an interface applications use to connect to data sources, execute SQL, and retrieve results. The usual request path is application → driver manager → source-specific driver → data source. ODBC standardizes application calls; it is not a universal database network protocol.

Federated queries let Amazon Redshift query supported operational databases without first loading their tables into Redshift. Supported sources include Amazon RDS for PostgreSQL and MySQL, and Aurora PostgreSQL-Compatible and MySQL-Compatible databases. This is not a connection mechanism for every database engine.

A join combines related records by evaluating a matching condition. Joining orders to customers on customer_id adds customer attributes to each matching order, making revenue by customer region possible. Unlike UNION, which combines result rows, a join matches records and brings their columns together.

Deduplication removes redundant representations of the same thing. Overlapping sources and repeated deliveries can create those copies. First define the record grain: does one output row represent a customer, a payment event, or a particular version of a payment? Two payments by the same customer are not duplicates merely because they share a customer identifier.

A data contract defines more than field names and types. It also states what values mean, which fields may be absent, and how readers should interpret them. Schema evolution changes this contract while keeping supported records and applications usable.

A processing job can become cheaper per attempt but more expensive to operate if it fails and retries more often. Use cost per successful run: total processing cost over a representative period divided by the number of runs that produced correct, complete results. Include failed attempts and retries in the cost, not in the successful-run count.

Incremental processing starts with an initial result and maintains it as the source changes. A sales pipeline can keep yesterday’s totals, process newly arrived transactions, and update only the affected totals instead of reading and aggregating all historical transactions again.

Amazon Athena lets you query data directly in Amazon S3 with SQL, without first loading it into warehouse tables. Table metadata describes how to interpret the files. AWS manages the query infrastructure, making Athena useful for exploring data already stored in S3.

Amazon EMR and AWS Glue can both run large-scale data transformations using Apache Spark. Cleaning records, joining datasets, and producing analytical files are not exclusive to either service. Data size alone does not tell you which service to choose.

AWS Glue DataBrew fits preparation that people can express through visual transformations. You inspect data, select operations such as trimming text or converting column types, and record them in a recipe. This lets people who understand the data maintain preparation logic without writing an ETL script.

Amazon EMR runs data-processing frameworks such as Apache Spark. For a large log archive, distributed processing means dividing the data and computation across workers instead of processing everything on one machine. This lesson uses EMR on EC2.

For standard on-demand Lambda event processing, each execution environment handles one invocation at a time. Memory allocation applies to each environment, not to a shared pool for the whole function. Increasing memory also increases the CPU resources available to that environment.

A SQL transformation turns source rows into a useful analytical result. In Amazon Redshift, WHERE filters rows, expressions calculate values, joins combine tables, and aggregates such as SUM summarize them. A SELECT returns a result; it does not, by itself, create a stored table.

Data compression reduces the bytes needed to represent information. For analytical records, use lossless compression: decompression restores exactly what was compressed. For example, compressing a CSV file with gzip produces a compressed byte stream; decompressing it restores the original CSV bytes. The records still have a row-oriented text layout.

Converting CSV to Parquet starts by deciding what each field means. CSV supplies text fields, not a reliable stored type for each column. Define an explicit schema: for example, order_id as a string, amount as decimal with precision 12 and scale 2, and sold_at as a timestamp. Keeping the identifier 0012 as a string preserves its leading zeros; treating it as an integer does not.

Amazon Kinesis Data Firehose, now called Amazon Data Firehose, can convert incoming JSON records into Apache Parquet or Apache ORC before delivering them to Amazon S3. These columnar formats let analytical queries read selected columns instead of reading every field in every record.

Data skew means that a few partitions receive much more data or processing work than their peers. A partition is a logical slice of a distributed dataset; tasks process those slices in parallel. When most tasks finish but one overloaded task continues, that straggler determines when the stage finishes. More workers do not automatically divide its work.

A schema-related failure often means that two descriptions of the same data disagree. Compare four things: incoming records, the physical file schema, the catalog definition, and the job's expectations. Check field names, types, nested structure, and whether missing values are allowed.

Amazon EMR status tells you where to start investigating, not necessarily why processing failed. For Spark submitted as an EMR step on an EC2 cluster, inspect the step state and failure details. A FAILED step means the submitted work failed; a cluster in WAITING can still be healthy and ready for more work. Cluster health and application success are different things.

A failed or slow AWS Glue Spark job run needs two kinds of evidence. Logs explain events: exceptions, failed tasks, and messages from your script. Execution metrics show how work progresses: task duration, data processed, memory use, and shuffle activity. A shuffle redistributes data between tasks, often for a join or aggregation.

Amazon API Gateway exposes an HTTP interface for a backend. A request's method and path select an operation, and its integration defines how API Gateway calls the backend. HTTP APIs call these selections routes; REST APIs configure methods on resources.

For an API Gateway REST API, a deployment is a snapshot of the API configuration, including its resources, methods, and integrations. Editing that configuration does not change the snapshot already serving clients. You create another deployment to publish those API changes.

A pagination contract defines how an API divides a collection into bounded pages. Choose a default page size, enforce a maximum, and document how invalid sizes are handled. For example, your API can reject sizes above its maximum rather than silently reducing them. A requested size is an upper bound on returned records, not a promise that every page will contain that many.

A data API contract defines an operation independently of its storage implementation. For this lesson, implement GET /v1/orders/{orderId}. The caller supplies a nonempty identifier containing only letters and digits. A successful response is HTTP 200 with JSON such as {"orderId":"A17","status":"SHIPPED"}.

Structured data uses a consistent schema: a definition of fields, their meanings, and their data types. A record supplies values for those fields. In a table, each row is a record and each column is a field. Records share the definition, not necessarily the same values.

Semi-structured data organizes information within each record using names and relationships. In JSON, a named field pairs a key with a value. For example, in {"event_id":"p17","customer":{"id":"c8"},"items":[{"sku":"A1","quantity":2}]}, the key event_id identifies the event. A reader looks up that name rather than relying on its position in the object.

Unstructured data includes free-form text, images, audio, and video. Its meaningful content is not arranged as a predefined set of fields for analysis. These files still have technical formats: a photograph has pixel values, and an audio recording has a sample rate.

Structured data uses a consistent schema. In a purchase table, every row follows defined columns such as event_id, amount, and coupon_code, with an agreed type for each column. A nullable coupon column can contain null for some purchases without changing the table's layout: the column still belongs to the shared schema.

Structured data represents information through an explicit, consistent schema: fields with defined meanings and data types. An order record might contain order_id, amount, and currency. Software can total the amounts or group orders by currency using those definitions, without interpreting arbitrary prose.

Semi-structured data organizes records with named fields, tags, or nesting. A JSON event with an identifier, a timestamp, and an array of items gives software recognizable places to find values. Records can have different optional fields and still have useful organization.

A large language model (LLM) is a type of broadly pretrained foundation model. During inference, it uses a prompt to generate an answer without changing its model weights. For data transformation, the input might be a support note, email, or product description; the output might be a JSON record with named fields. This is useful when wording varies enough that fixed extraction rules become difficult to maintain.

Amazon Bedrock provides managed access to foundation models: models already trained for broad tasks such as summarization, text generation, and embeddings. Your application selects a suitable model and calls an API. AWS operates the underlying inference infrastructure, so your team does not need to build a model-serving environment just to use that model.

A prompt template is a reusable specification for constructing an LLM request. It combines fixed instructions, which define the transformation, with runtime variables, which supply the values for one request. The application fills the variables before calling the model. This lets a data-processing job apply the same extraction or summarization rules to many records without rewriting those rules each time.

Foundation model inference turns application input into generated output: for example, a support message into a summary or a product description into category labels. Amazon Bedrock operates the inference infrastructure; your application selects the model and constructs the request. Evaluate candidates using representative records for output quality, supported inputs, context capacity, latency, and cost. Check availability in the intended AWS Region.

An LLM response is untrusted input to the next component. Asking for JSON, or using structured generation, does not establish that its values are safe or correct. Define an application-owned contract and check the response before downstream processing.

Knowledge Bases for Amazon Bedrock connects your documents to model requests through retrieval. For a typical text-document workflow, ingestion reads a source such as Amazon S3, extracts its content, and divides it into chunks: passages that can be retrieved separately.

A state machine is a workflow definition written in Amazon States Language. An execution is one run of that definition with its own input and progress. A Task state invokes work, such as a Lambda function or an AWS Glue job; Step Functions coordinates that work rather than performing the transformation itself.

A failed task does not always need to end a workflow. In AWS Step Functions, Retry repeats work after a matching error, while Catch sends control to an error-handling state. For states that support both, Step Functions evaluates matching retry rules before catch rules. Catch rules are considered when no retry applies or the matching retry policy is exhausted.

Declare a Step Functions workflow with AWS::Serverless::StateMachine in an AWS SAM template. SAM transforms this declaration into CloudFormation resources, so the workflow can be deployed alongside the functions and tables it uses. Include the SAM Transform at the template's top level.

Amazon MWAA runs Apache Airflow workflows. An Airflow directed acyclic graph (DAG) describes tasks and their dependencies, commonly in Python. For an extract → transform → load pipeline, dependencies say that transformation must wait for extraction, and loading must wait for transformation. The graph cannot contain a dependency cycle; repeated processing happens through separate runs or task logic, not an edge back to an earlier task.

High performance means meeting a workload's speed and processing-capacity targets. Throughput measures how much work completes in a period; latency measures how long work takes. A pipeline can process many records overall while some individual records still take too long.

High availability focuses on keeping a service usable and minimizing interruptions. Resiliency is broader: it is the ability to withstand disruptions and recover acceptable service afterward. A resilient system may keep working normally, offer reduced functionality, or temporarily stop and recover within agreed limits.

Data durability concerns preserving stored data against permanent loss. Availability concerns whether a service can perform the reads or writes a workload needs when it needs them. High availability aims to keep that access working through disruptions.

Throughput is successful work completed per unit of time. Latency is the time one item spends travelling through the pipeline. A pipeline can complete many records each second while some records wait a long time. Neither measure establishes the other.

High availability keeps a service usable when components fail. Redundancy provides another component that can do the work, but its placement matters. Two workers on different machines can survive one machine failing; if both occupy the same Availability Zone, a disruption of that zone can affect both.

A replica receives changes from a source database so another current copy is available. Place it in an independent failure domain: a different Availability Zone protects against a zone failure, while regional disruption requires a recovery strategy outside the affected Region. Another copy beside the source does not provide the same isolation.

An alert-producing application calls SendMessage to place an alert in an Amazon SQS queue. SQS stores the message independently of the producer and consumer, so the producer does not have to wait for a worker to process it. This example uses a Standard queue.

When every pipeline stage waits for the next stage, one slow dependency can tie up the whole chain. A durable processing boundary stores accepted work independently of the process that will handle it. A worker can stop without making the producer repeat all earlier work.

Serverless means the provider operates the servers, not that servers disappear. With an application on self-managed Amazon EC2 instances, your team chooses instance capacity, maintains the guest operating system, and arranges instance replacement and scaling. With serverless execution, AWS provisions and maintains the underlying hosts. You work with higher-level resources such as functions or container tasks.

A reliable serverless workflow separates durable coordination from bounded execution. AWS Step Functions Standard workflows retain execution state between steps. AWS Lambda runs the code for a step, but its execution environment can be replaced. A later invocation must not depend on an earlier invocation's memory or local files.

Serverless answers an infrastructure question: who operates the servers? The provider provisions, maintains, and replaces the underlying hosts. Your team works with resources such as functions or managed container tasks instead of administering those hosts. You still own application code, permissions, data, and configuration.

An alert producer can submit work to Amazon SQS instead of waiting for an alert worker. The queue retains accepted messages while workers are busy or unavailable, subject to its configured retention period. This separates the producer's availability from the worker's availability.

An Amazon SNS topic is a named destination that publishers send messages to. A subscription connects the topic to an endpoint. For monitoring alerts, a Standard topic can distribute one published message to several endpoints without the publisher contacting each one.

A monitoring alert can require two different jobs: deciding which events should trigger which responses, and distributing an already selected notification. Amazon EventBridge uses rules on an event bus to match event content and route matching events to targets. Choose it when monitoring events need different responses based on their source, event type, or details.

Vectorization applies an operation through an array, column, or batch interface instead of explicitly handling each record in application code. First identify the bottleneck: a CPU-heavy Python loop is a good candidate; slow source reads or network transfers are not fixed by changing arithmetic. Native operations reduce repeated interpretation, function calls, and object handling. They do not necessarily use hardware vector instructions or add workers.

An AWS CLI script turns a manual service call into a repeatable operation. The usual shape is aws service operation --parameter value. For example, aws glue get-job --job-name daily-etl --profile data --region us-east-1 --no-cli-pager retrieves a Glue job definition. --job-name is an operation parameter; --profile and --region select the credential configuration and Region.

Serialization turns an in-memory record into a representation that can be stored or sent to another program. Deserialization interprets that representation as usable values. The receiving program does not need the same programming language, but it does need compatible rules.

An exception interrupts normal execution and travels outward until a matching handler catches it. A failure boundary is the unit of work that a handler protects: one field conversion, one record, one dependency call, or the whole task. Choose that boundary according to what can safely continue after the failure.

A Python generator produces values when a consumer requests them. Calling a function that contains yield creates a generator iterator; it does not run the function body yet. This lazy evaluation lets a pipeline start processing records without first preparing every result.

Amazon Athena runs SQL over data in Amazon S3 without requiring you to load that data into a separate database. For a cataloged S3 table, Athena uses the AWS Glue Data Catalog to find its column definitions, file format, and S3 location. The catalog stores metadata, not the table's rows.

A join combines rows from two inputs, such as tables or subqueries. The ON condition decides which row pairs match. For example, FROM orders o INNER JOIN customers c ON o.customer_id = c.customer_id pairs each order with every customer row having the same customer ID. A declared foreign key is not required.

A window function calculates across related rows while preserving each row supplied to it. Compare SELECT customer_id, SUM(amount) FROM transactions GROUP BY customer_id, which returns one row per customer, with SELECT customer_id, amount, SUM(amount) OVER (PARTITION BY customer_id) AS customer_total FROM transactions, which keeps every transaction and adds its customer's total.

In Athena, cleaning with SELECT means calculating new values or filtering rows in query results. It does not repair or overwrite the source objects in Amazon S3. We will use orders_raw, with text columns order_id and amount_text, a timestamp updated_at, and a unique numeric ingest_id.

Amazon Athena queries log files in Amazon S3 without loading them into a separate database. A log-table definition describes how to interpret those files as rows and columns. This is schema on read: Athena applies the definition when it reads the data; creating the table does not rewrite the logs.

Amazon Athena for Apache Spark lets you explore data interactively with PySpark, the Python interface to Spark. Athena provides the Spark compute: you do not provision cluster nodes, install Spark, or maintain a cluster.

Git preserves identifiable revisions of files in a repository. For a data pipeline, track transformation scripts, SQL, workflow definitions, infrastructure templates, and tests together. A commit records a snapshot of tracked files, its parent history, author information, and a message. Its commit ID identifies that revision; a useful message explains why the pipeline changed.

AWS CodeCommit hosts Git repositories without requiring your team to operate a Git server. A data engineering repository can hold transformation scripts, SQL models, workflow definitions, infrastructure code, and tests. Keep credentials and production datasets out of the repository; version the code that processes data, not the data itself.

Static analysis inspects code or configuration without running the data pipeline or deploying the infrastructure it describes. The analyzer itself runs, but the pipeline remains an artifact to inspect. Inputs can include transformation scripts, SQL files, infrastructure templates, identity policies, and job configuration.

A transformation's public contract describes what callers can observe: accepted inputs, returned data, errors, and promised effects. Unit tests protect that contract rather than the sequence of private helpers used to implement it. This lets you improve the implementation without rewriting tests for behavior that has not changed.

Unit testing checks a coherent piece of behavior independently of its wider dependencies. Integration testing checks whether connected components work together. The distinction is the test boundary: which implementations participate, and which dependencies are kept outside the check.

An integration test checks whether connected components honor their shared contract. Here, an AWS Glue job reads an Amazon S3 fixture and writes transformed records to another S3 location. The contract covers what the job reads, what it writes, and how it treats invalid input—not just whether it finishes.

Integration testing asks whether selected, connected pipeline components work together correctly. End-to-end testing asks whether the complete defined pipeline produces the intended outcome, from its normal entry point to its final destination.

Structured logging gives each pipeline event named fields with defined meanings. A readable message can help a person, but separate fields let tools filter and aggregate without interpreting prose. JSON is a useful encoding; JSON alone is not a contract.

AWS CloudTrail records AWS API activity and related account events. Its records help answer who did what, when, and from where. Calls made through the console, command-line tools, SDKs, and AWS services can produce this audit evidence.

CloudWatch Logs stores application log events centrally, independently of the host that produced them. An EC2 application does not send its local files automatically: a collector such as the CloudWatch agent must read and publish them.

AWS CodeBuild runs build and test commands in a managed build environment. For a data pipeline, those commands can test transformations against small fixtures: for example, checking that duplicate rows are removed or invalid dates are rejected.

AWS CodePipeline coordinates a software release through stages and actions. It defines which work happens next and connects the services that perform that work. For a data pipeline, this could mean moving a changed transformation script through testing and then deployment.

Infrastructure as code (IaC) describes infrastructure in files that a deployment engine uses to create and update resources. For a data engineering environment, a definition could describe an S3 bucket, an AWS Glue job, and the job's access role. Store these files in version control so changes can be reviewed instead of repeating console operations.

AWS CloudFormation provisions infrastructure from a JSON or YAML template. Its Resources section declares resource types and properties rather than a sequence of API calls. Store the template in version control so deployments use a known, reviewable definition.

Infrastructure as code (IaC) records the infrastructure you want in files that can be reviewed, version-controlled, and reused. AWS CDK and directly written CloudFormation templates both support this approach. The main difference is how you author the infrastructure definition, not which service ultimately provisions it.

An AWS CDK app contains one or more stacks. A stack is a deployment unit that becomes an AWS CloudFormation stack. Inside it, constructs define resources and their configuration. A construct can represent one resource or package several resources into a reusable component.

AWS CloudFormation provisions and manages infrastructure described in YAML or JSON templates. Its resource types cover networking, compute, storage, and many other services. Related resources are managed together as a stack. CloudFormation can deploy serverless applications without SAM.

An AWS SAM template describes the resources and configuration for a serverless application. It includes Transform: AWS::Serverless-2016-10-31 and a Resources section. Deploying it creates or updates a CloudFormation stack: the resources managed together for this application.

AWS Lambda ephemeral storage is temporary filesystem space available at /tmp inside a function's execution environment. Lambda provides it already mounted: your code creates directories and reads and writes files using ordinary filesystem operations. You do not attach a volume or run a mount command.

A Lambda function can use Amazon EFS when its code needs persistent shared files through ordinary file operations. Configure the function with an EFS access point and a local mount path such as /mnt/shared. Lambda mounts the file system as it prepares the execution environment; your handler opens paths rather than running a mount command.

Amazon EFS gives Lambda functions a persistent, shared file system. Your code uses ordinary file operations through a local mount path, such as /mnt/shared. Files remain in EFS after an invocation ends, and other authorized functions can access them.

A Lambda function can use both local ephemeral storage at /tmp and a shared file system mounted from Amazon EFS. Both expose files and directories, but a path alone does not tell you who can see its files.

Configuration drift is a difference between the configuration you expect and the configuration that actually exists. For infrastructure as code, the baseline is the approved resource definition, together with the deployment inputs such as parameter values. Keeping the source and inputs in version control makes the intended configuration reviewable and reproducible.

Continuous integration (CI) combines frequent, small changes in a shared repository with automated build and test feedback. For a data workload, version the transformation scripts, tests, dependency declarations, and deployment definitions together. A scheduled build alone is not effective CI if developers leave incompatible changes separate for long periods.

An end-to-end test submits realistic records at a pipeline's normal entry point and checks the data that reaches its destination. A job reporting success is not enough: it can finish while writing incorrect totals, missing records, or the wrong schema.

AWS CodePipeline coordinates a release workflow. For a data pipeline, that release can contain transformation scripts, SQL, tests, and infrastructure templates. It promotes a software revision toward deployment; it does not itself process each incoming data record.

AWS CodeDeploy releases an application revision to a chosen environment. A CodeDeploy application organizes deployments for one compute platform: EC2/On-Premises, Amazon ECS, or AWS Lambda. Within that application, a deployment group identifies the deployment targets and settings. For EC2, targets can be selected using instance tags or Auto Scaling groups.

AWS CodePipeline coordinates a software release across stages such as source, build, test, and deployment. It decides which configured action runs next and uses that action's result to control progression. The services integrated with those actions perform the work.

Distributed computing divides a workload into tasks that run on multiple workers. Each worker processes part of the input, often called a partition. Parallel work can shorten processing time, but reading input and exchanging intermediate results can make the network a bottleneck.

Distributed processing divides a job into tasks that workers can run across multiple machines. Fault tolerance lets the job continue when a worker disappears: unfinished work moves to another worker instead of depending on the failed machine.

A rooted tree represents records arranged under one starting record, the root. Every other record has exactly one parent and can have children of its own. A node with no children is a leaf. A node and all its descendants form a subtree, which you can process as one part of the hierarchy.

A graph represents entities as nodes and relationships as edges. A traversal starts at a node and follows selected edges to discover other nodes. An adjacency list supports this by recording each node's neighbors, so traversal can find the next connections without scanning every entity.

A graph represents entities as nodes and relationships as edges. A tree is a special kind of graph: it is connected, meaning every node can be reached from every other node, and acyclic, meaning it has no cycles.

Algorithmic complexity describes how resource requirements change as an input grows. Before writing a formula, define the input size and the resource being counted. For a transformation that does a fixed amount of work per record, n can mean the number of records. For a task that examines every byte in variable-length files, total bytes may be the more useful measure: equal record counts can contain very different amounts of data.

A full tree traversal visits all n nodes. With constant work per node and child link, its time is O(n), whether the tree is balanced or skewed. A balanced tree does not make a full traversal logarithmic: it reduces the longest path, not the number of nodes to visit.

A technical catalog entry tells a query engine where a dataset is and how to interpret it. For a file-based external table, creating the entry does not load the records into the catalog. The files remain at their source.

Schema on read means applying a table definition when consuming stored data. Files can arrive in Amazon S3 before anyone defines the columns that an analytical query will use. Unlike schema on write, storing these files does not itself enforce that analytical definition.

For a conventional partitioned external table, storing files and describing them are separate jobs. A prefix such as s3://analytics/events/event_date=2026-04-15/ holds objects. Its name alone does not guarantee that Athena knows the partition exists.

For a traditional external table using registered partitions, the catalog connects each partition's key values to its storage location. For example, the keys event_date and region might map the values 2026-01-15 and eu to s3://lake/events/event_date=2026-01-15/region=eu/. The table defines the keys; each partition entry supplies values and a location.

A VPC endpoint lets a workload connect to a supported service without using an internet gateway or NAT device for that connection. Choose the endpoint type supported by the service and the access pattern you need.

AWS Secrets Manager stores sensitive values such as database usernames and passwords, API keys, and access tokens. A secret has an identifier, such as a name or ARN, and a protected value. Related fields can be stored together as a JSON string.

A business data catalog helps people find datasets and understand what they mean without first opening the data. Business catalogs such as Amazon SageMaker Catalog support this discovery and governance work. The catalog describes assets; the underlying data can remain in its storage system.

Amazon DataZone provides a business catalog: descriptions of data that help people find and understand it. A domain is the organizational boundary containing projects, catalog assets, glossaries, and metadata forms. The catalog describes data; it does not replace the systems that store it.

Amazon SageMaker Catalog helps people discover organizational data and understand whether it fits their work. A catalog asset represents a resource, such as a table, through metadata. The catalog entry describes the resource; it is not a copy of the table's records.

In Amazon SageMaker Unified Studio, a domain brings together users, projects, and governance settings for an organization’s data and analytics work. It provides the organizational context for collaboration, not permission to read every dataset within it.

S3 Lifecycle applies actions to objects that match a bucket's lifecycle rules. A transition changes an object's storage class; expiration deletes data according to the bucket's versioning behavior. A transition does not require expiration. Age-based transitions do not measure access: reading an object does not reset its lifecycle age.

S3 Lifecycle automates storage-class changes for objects in an S3 bucket. A transition into a Glacier storage class keeps the object in the same bucket with the same key. It does not move the object into a separate Glacier vault.

A retention policy needs both a minimum retention period, before which data must not be destroyed, and any disposal deadline, after which it should no longer remain. Identify the data owner, the records covered, the event that starts the clock, and approved exceptions before configuring deletion.

Data retention defines which data to keep, how long to keep it, and what should happen afterward. A usable policy identifies the data owner, the reason for retention, and the event that starts the clock, such as record creation or account closure. These periods come from applicable business and legal requirements, not universal cloud defaults.

A recovery point is a backup you can restore from. AWS Backup stores recovery points in backup vaults. A backup rule's lifecycle determines how long its recovery points are retained and, for supported resources, when they move to cold storage.

A legal hold prevents permanent deletion of information needed for a legal matter or investigation until an authorized release. Legal or records-management teams define what must be preserved; engineers apply and verify the controls. A case tag helps locate evidence, but does not itself block deletion.

Data durability is protection against permanent data loss. Storage systems use redundant copies or erasure coding, which spreads encoded fragments so data can be reconstructed after some fragments are lost. Integrity checks detect damaged data, and repair restores redundancy while enough healthy data remains.

Normalization places facts in tables according to what determines them. Repeating a customer's name on every order line creates an update anomaly: a rename requires several changes. An insertion anomaly prevents recording a customer without an order line. A deletion anomaly loses customer details when the last related line is deleted. Removing duplicate rows alone does not fix these problems.

A read-oriented record stores the values needed by a particular query together. Start by writing its access pattern: which records are selected, how they are ordered, and which attributes the result needs. Related data is not automatically useful data.

Normalization organizes relational data so that each kind of fact is maintained in its proper place, reducing unnecessary duplication. For example, Customers(customer_id, current_name) stores a customer's current name, while Orders(order_id, customer_id) refers to that customer. Many orders can repeat the customer identifier without repeating the customer's name. Normalization does not mean eliminating every repeated value.

A star schema organizes an analytical model around a business process. Its central fact table stores measurements and keys referencing dimension tables. Dimensions supply the descriptions used to filter and group those measurements: customer region, product category, or calendar month.

A star schema is an analytical structure: a central fact table holds measurements and keys that reference surrounding dimension tables. For example, Sales can hold quantity, sales amount, product key, and customer key. Product and Customer hold descriptions used to filter and group those measurements. A report joins sales to products to total sales by category.

A dimension describes business entities such as customers or products. Slowly Changing Dimensions (SCD) addresses changes to those entities’ descriptive attribute values. If customer C42 moves from East to West, the value of region changes, but the table can still have exactly the same columns and types.

A slowly changing dimension (SCD) policy decides what happens when a business entity’s descriptive attributes change. Choose it from the reporting need: should past sales show a customer’s original region, latest region, or region when each sale happened?

Backward schema compatibility means a new reader can correctly interpret records written with an older writer schema. The reader changes; the historical records stay unchanged. This matters when a new application replays old events or queries files written before its release.

A heterogeneous migration moves a database between different engines, such as Oracle and PostgreSQL. SQL support differs between engines, so copying rows does not recreate the full database. Schema conversion translates object definitions, including tables, views, constraints, and supported database code.

Schema conversion translates database definitions and supported database code for a different engine. It is separate from copying table rows. Both DMS Schema Conversion and the AWS Schema Conversion Tool (AWS SCT) help with this work, but they run in different environments.

AWS Schema Conversion Tool (AWS SCT) is a downloadable tool that translates database definitions between supported database engines. It can convert tables, views, and database code such as stored procedures. Conversion produces target-engine SQL; it does not prove that the converted code behaves correctly or that table rows have been migrated.

DMS Schema Conversion helps prepare a database schema for a different database engine. It works with object definitions and database code, not the rows stored in tables. Converting a schema and migrating its data are separate jobs.

Data provenance is the evidence behind a dataset’s history. Lineage connects the sources, transformations, and outputs in that history. A useful record lets you start with an output and find the exact inputs and processing run that produced it.

Lineage records where data came from, which operations used or changed it, and what those operations produced. A list of datasets and models is not enough: the relationships explain how a source dataset contributed to a prepared dataset and, later, a model.

An index is an alternative access path to data. Start with observed queries: their filters, join columns, requested order, returned columns, frequency, and latency needs. A lookup returning a few orders has different needs from a report reading most of the table. A selective predicate matches a small fraction of rows; it often makes an index more useful.

A lossless codec compresses data so that decompression restores the original bytes exactly. This is the normal choice for analytical records, where changing a value is unacceptable. The reduction depends on the input: repetitive text often compresses well, while encrypted or already compressed data usually offers little additional saving.

An embedding is a learned numerical representation of an input. For document retrieval, an embedding model converts a passage or query into a vector: an ordered list of numbers. The output has a fixed number of positions, called dimensions, for the chosen model and output configuration.

Embedding production uses a trained machine learning model to turn an input into a numerical representation. For text retrieval, that representation captures learned characteristics of a passage or query. The output is a vector, not a list of matching documents.

A document-based knowledge base prepares source material before answering questions. During ingestion, it parses documents into text and divides that text into chunks: smaller passages that can be retrieved individually. Each chunk should preserve enough meaning to be useful without bringing an entire document into the model's context.

Vector search finds records whose numerical representations are close to a query's representation. Its output is matching records, often with similarity scores. A search for account recovery might return a password-reset passage even when the wording differs.

Amazon RDS provides managed relational databases commonly used for operational workloads: the reads and writes that keep an application running. Creating an order, changing a delivery address, and retrieving one customer's purchases usually touch a small, selected set of records. These workloads are often called online transaction processing (OLTP). They prioritize quick responses for many short operations and correct changes to current data.

A database on Amazon EC2 is self-managed: AWS supplies the virtual infrastructure, and you install, configure, and operate the database software. Amazon RDS supplies a managed relational database instead. In both cases, AWS operates the underlying physical infrastructure; the difference is how much of the software stack you operate.

Amazon RDS runs relational databases: data is organized into tables of rows and columns. A primary key identifies a row, and a foreign key can refer to a row in another table. For example, an Orders table can store a customer ID that links each order to its customer. Many-to-many relationships commonly use an association table containing pairs of IDs.

An access pattern describes what the application must retrieve and which values it knows when making that request. For an orders application, examples are “list one customer's orders by date” and “list orders with a particular status.” Design keys around these reads, rather than storing attributes first and expecting to search any of them efficiently later.

Amazon Redshift stores table data by column. A query that sums sales amounts does not need to read unrelated columns such as delivery instructions. Selecting only needed columns preserves this advantage; SELECT * can waste it. Massively parallel processing divides work among compute resources, so good table design must also spread work evenly.

Data partitioning divides records into separately managed subsets according to a rule. For example, an orders table can have one partition for each month of order dates. The partitions still belong to one logical dataset; they do not have to run on different servers. Here, partitioning means dividing rows, not separating groups of columns.

Data partitioning organizes records into subsets according to a rule. For example, event files can be grouped under events/event_date=2026-01-15/ and events/event_date=2026-01-16/. Each location contains data for one date. That organization exists independently of any query.

Data partitioning divides records into subsets using a rule. Choose that rule from the access pattern: which filters appear most often, how much data each request needs, and whether requests target particular customers or time windows. A useful partition key lets common reads identify a small part of the dataset without making writes concentrate in one place.

Time to Live (TTL) lets DynamoDB remove items after a timestamp you store on each item. Enable TTL on the table and select one attribute name, such as expiresAt. In the console, use the table's TTL settings; through the API or SDK, use UpdateTimeToLive with Enabled=true and AttributeName=expiresAt. Enabling TTL is asynchronous, so check its status with DescribeTimeToLive rather than assuming it is immediately active.

Amazon DataZone helps people discover, understand, and request organizational data. Consumers search a business catalog, review descriptions and ownership information, and request subscriptions to published assets. This answers: Which data should I use, and who should approve my use?

A key-value model associates a unique key with an application value. Start by asking what information a request already contains. If a cart request supplies a globally unique customer ID, the application can construct cart:customer-482 directly. It does not need to search stored carts to discover their keys.

Amazon MemoryDB for Redis is a managed database that combines Redis-compatible access with durable storage. It keeps data in memory for fast access, but it is not merely a temporary copy of data held in another database: it can serve as an application's primary database.

An embedding is a list of numbers produced by a model to represent an input. A model trained for semantic retrieval can place text with related meanings near each other. That lets “recover account access” find a passage about resetting a password even when the words differ.

A nearest-neighbor query asks for the top k stored vectors closest to a query vector under a chosen distance measure. An exact search returns the true nearest neighbors under that measure. A brute-force search compares the query with every eligible vector, which can become expensive as the collection grows.

Amazon Aurora PostgreSQL can use the pgvector extension to store and compare embeddings. An embedding is a list of numbers produced by a machine learning model to represent content, such as a product description. Comparing these vectors helps an application find related content even when the words differ.

AWS Transfer Family provides managed file-transfer endpoints for existing SFTP, FTPS, and FTP clients. You configure the endpoint, users, and storage access instead of maintaining file-server software. SFTP and FTPS encrypt traffic in transit; FTP does not.

AWS DataSync moves files and objects between supported storage systems. A task connects a source location to a destination location and defines how transfers run. Starting that task creates a task execution: one run of the saved configuration.

AWS DataSync automates moving files and objects between supported storage locations. Choose it when your job is to migrate an existing dataset or repeatedly copy changes from source storage to destination storage.

Amazon Redshift federated queries read supported Amazon RDS and Aurora PostgreSQL or MySQL databases without first loading their tables into Redshift. An external schema maps a remote database namespace into Redshift SQL. You can join its tables with local warehouse tables; creating the schema does not copy the remote rows.

Concurrency control lets operations overlap without breaking data rules. A database lock reserves access to a resource so that conflicting operations cannot proceed together. It is not authorization: permissions decide who may access data; locks coordinate operations that are already allowed.

Transaction isolation controls how concurrent transactions affect one another. A lock protects a database resource while work is in progress. Locks coordinate operations; they do not replace database permissions or IAM authorization.

A database lock coordinates concurrent access to data. Suppose transaction A holds an exclusive lock on account 100, while transaction B holds one on account 200. A requests 200 and must wait for B. If B then requests 100, it must wait for A. This circular wait is a deadlock: neither transaction can finish and release its locks without intervention.

In Amazon RDS, the database engine manages transaction locks, not the RDS console or IAM. A lock coordinates concurrent access to a database resource, such as a row or table. When two requested lock modes conflict, one transaction must wait. Locks protect data consistency; they are not a replacement for database permissions.

A transaction groups database work into one unit: either its changes are committed or they are rolled back. In Amazon Redshift, BEGIN; starts an explicit transaction, COMMIT; makes its changes permanent, and ROLLBACK; discards its uncommitted changes. With autocommit enabled, statements outside an explicit transaction normally commit individually; check your client's transaction settings.

An Apache Iceberg table is more than a directory of data files. A snapshot describes a particular table state through metadata that identifies its data files and, where applicable, delete files. Readers discover the table through its metadata, rather than treating every file in a storage location as committed data.

Optimistic concurrency control lets Iceberg writers prepare changes concurrently rather than hold an exclusive lock throughout processing. Each writer starts from a table version and prepares new files. Preparing files is not the same as committing them.

A data lake has several descriptions of its data: the physical schema in files, the catalog schema describing a table, and the schema a reader expects. An AWS Glue crawler discovers schemas and populates the AWS Glue Data Catalog. Updating the catalog does not rewrite historical files or backfill values into old records.

Hierarchical Navigable Small World (HNSW) is an approximate nearest-neighbor index. It represents indexed vectors as graph nodes and stores links between them. Instead of comparing a query with every vector, search follows links toward promising candidates, often saving substantial work.

An inverted file index (IVF) organizes vectors into lists so a similarity query can search a subset of the data. Unlike a text inverted index, which maps words to document identifiers, IVF maps regions of vector space to lists of vectors. The embedding model creates the vectors; the index organizes them for retrieval.

IVF, short for inverted file, and HNSW, short for Hierarchical Navigable Small World, organize vectors for approximate nearest-neighbor search. They reduce search work but can miss true nearest neighbors. Here, IVF means a vector index with partitioned lists, not a keyword inverted index.

In Amazon MWAA, start troubleshooting in the Airflow UI with the affected DAG run and its task instances. A task instance belongs to a particular run, and retries produce separate attempts. Record the DAG, run, task, attempt, and time before reading logs. A DAG run's overall status does not identify which task caused the problem.

An AWS SDK service client exposes operations for one AWS service. Two independent settings determine how it calls that service: credentials identify the caller, while the Region helps select the service endpoint. Valid credentials do not grant every permission, and choosing a Region does not change the caller's identity.

An AWS SDK is a language-specific library for calling service APIs. Application code creates a service client with a Region and suitable timeout and retry settings. For example, Python's Boto3 can create an S3 client with s3 = boto3.client('s3', region_name='us-east-1'). The SDK builds HTTP requests, signs them, and converts responses into language-native values.

An application can call AWS STS AssumeRole through an AWS SDK to obtain temporary credentials for an IAM role. It starts with source credentials that identify the caller; assuming a role is not an anonymous request.

Credential rotation must update both the credential accepted by the target system and the secret applications retrieve. Changing only the stored password does not change the database user's password.

AWS Secrets Manager and AWS Systems Manager Parameter Store can both store credentials outside application code. Both support encryption with AWS KMS, access control through IAM, and API activity logging through AWS CloudTrail. Encryption alone is not a reason to choose Secrets Manager.

An API contract defines what a consumer sends and receives: endpoints, parameters, fields, types, authentication, errors, and behavior such as pagination. An ingestion job depends on that whole agreement, not just a response that parses successfully. A contract version is distinct from an SDK package version, an implementation deployment, or an environment named production.

Data profiling discovers what is present in a dataset. It measures observed characteristics, such as missing-value percentages, distinct values, and date formats. It answers, “What does this data contain?” rather than assuming that documentation describes the actual records.

A dataset profile summarizes the records actually inspected, rather than just their declared schema. Start with its scope: which files, partitions, time period, and rows were included? A sampled profile describes the sample; it can miss rare defects. Record the sample size and selection method, and use full or targeted checks for critical constraints. AWS Glue DataBrew profile jobs provide statistics for investigating data before preparation.

AWS Glue DataBrew makes data quality visible through column statistics and distributions. Missing-value statistics reveal gaps, distinct-value counts show variety, and value frequencies or numeric distributions show how values are spread. A profile job calculates dataset and column statistics independently of an interactive project session.

AWS Glue DataBrew lets you prepare data visually. A dataset identifies the input source and its format or connection settings. A project opens an interactive workspace on a sample of that dataset, so you can explore values and preview transformations without displaying every source record.

An Amazon SageMaker Unified Studio notebook combines executable code, results, and notes in a project workspace. Use it to explore project data and develop repeatable preparation steps. Data being available to a project does not mean it is already loaded into a notebook: a cell must read the source using an appropriate connection or library.

Amazon Athena can query S3 tables governed by AWS Lake Formation. This lesson follows a table whose S3 location is registered with Lake Formation and whose Lake Formation permissions are enforced, rather than relying on IAM-only defaults. Query results are stored in a separate, customer-managed S3 location.

Amazon OpenSearch Service is a strong choice when investigating logs means asking many follow-up questions: find an error message, narrow to one application, then inspect related events. In the indexed approach, logs are ingested into OpenSearch and indexed before exploration. That preparation supports repeated searches and aggregations through tools such as OpenSearch Dashboards.

In Amazon QuickSight, a dataset supplies the fields, an analysis is the author's workspace, and a dashboard is the published experience for readers. Start by selecting a dataset and creating an analysis. Check the field types before building visuals: dates should be dates, and amounts used in calculations should be numeric.

Data validation makes a dataset requirement executable. Start with an explicit expectation, its scope, and its acceptance condition: for example, every order in today's delivery must have a numeric, nonnegative amount, with zero violating rows. Profiling reveals patterns and suggests rules; it does not decide which values the business permits. An unusual amount is not automatically an invalid amount.

Deduplication removes redundant records, not every repeated value. First define the dataset's grain: what one row represents. For an order-line dataset, the intended business key might be (order_id, line_no). Two lines in the same order are legitimate repetitions of the order ID, not duplicate order lines.

Missing value imputation replaces an absent value so a record remains usable. Before choosing a replacement, identify the field’s business meaning. A missing package weight is not a measured weight of zero. A missing payment amount or order identifier may require source correction or quarantine rather than an invented value.

Amazon SageMaker Data Wrangler lets you build a visual data-preparation flow, including within SageMaker Canvas. Import a dataset, inspect its columns, and add transformations as steps. The flow records the preparation logic; adding a step does not overwrite the source data.

Elasticity means adding capacity as demand rises and releasing it as demand falls. Scalability describes the ability to handle more or less work; elasticity emphasizes matching capacity to changing demand. A fixed fleet of cloud servers is not elastic merely because more servers are available to purchase.

Usage-based pricing means paying for measured quantities, but the quantity is not always useful processing work. With provisioned capacity, you pay for capacity kept available over time. A running Amazon EC2 worker incurs compute charges even while it waits for its next data-processing job.

Aggregation combines values from several records into a summary. Choose the function to match the question: SUM(amount) gives a total, COUNT(*) counts records, and AVG(amount) gives the arithmetic mean of non-null amounts. MIN and MAX return the smallest and largest values. Counting records is not necessarily counting customers: one customer can have several records.

Lateral movement is the expansion of access after an initial foothold, such as a compromised application server reaching other internal hosts. A useful baseline describes expected activity for a particular source, identity, or workload role over comparable time periods. One fleet-wide total can hide a small source suddenly reaching unfamiliar systems.

An audit trail connects an actor to an action, target, time, source context, and outcome. Preserve the original identity fields: an assumed-role name alone may not identify the person behind a temporary session. Application audit records should also capture business actions and correlation identifiers without recording credentials or unnecessary sensitive payloads.

AWS Config answers questions about resource state: “What settings did this resource have?” and “What changed between recorded states?” Its central record is a configuration item, which describes a supported resource at a point in time, including configuration attributes, identifying metadata, and relationships with other resources where supported.

AWS CloudTrail records supported AWS API activity, including calls made through the console, CLI, SDKs, and AWS services. A trail selects events for ongoing delivery to Amazon S3. Selection matters: creating a trail does not mean every kind of activity is recorded.

CloudTrail Lake lets you retain activity events and investigate them with SQL. Its main storage resource is an event data store: a managed collection of immutable events with a configured retention period. You do not need to build a separate database or query pipeline for its native SQL queries.

Log retention specifies how long required records must remain available. Define both a searchable window for immediate investigation and a total retention period for older evidence. Also record the retrieval deadline for each age range: keeping a record is not enough if you cannot produce it in time.

An audit log export is evidence, not just input for a search. Preserve the original export exactly as collected, including its file format and compression. A filtered report or reformatted JSON file can be useful, but it is a derivative, not a replacement for the original.

Aggregation brings logs together. Event correlation links records that describe related activity. For a data pipeline, this can connect a run starting, an API call writing an object, and an audit record of that call. Collection alone does not establish those relationships.

Resource utilization measures how much available capacity a workload uses. A pipeline can have spare CPU while running out of memory or waiting for storage or network transfers. A bottleneck is the constraint that limits useful progress, so no single utilization percentage describes the pipeline.

Log normalization maps source-specific events into a shared schema so that a field has the same meaning across sources. A schema defines field names, types, event classes, and value conventions. It can be an established model, such as the Open Cybersecurity Schema Framework, or a documented model for your application.

Amazon OpenSearch Service stores log events as documents in an index. OpenSearch Dashboards lets you search those documents and visualize patterns. A field's mapping determines how OpenSearch interprets and indexes its values—not just how they appear on screen.

A data quality rule makes a consumer's requirement measurable. Start with what the consumer needs to do and what would go wrong with unsuitable data. An order lookup needs every order identifier, while a summary report may tolerate a small amount of missing optional information. There is no universal acceptable percentage.

Null handling starts with a field-specific data contract: which values mean missing, whether absence is allowed, and what to do when it occurs. A null represents absence, not automatically zero, false, an empty string, whitespace, or the text 'NULL'. JSON can also distinguish an absent property from a property explicitly set to null; decide whether that distinction matters before normalizing them.

Schema validation checks each record against an explicit contract before the processor treats it as valid. Successful JSON parsing only establishes valid syntax: {"attempts":"3"} parses successfully even when the contract requires an integer.

A processing boundary is a point where data enters a stage or becomes available to the next stage. Run checks during ingestion, after transformations, and before publication. An input check catches source problems; an output check catches problems introduced by processing.

AWS Glue Data Quality evaluates data against a ruleset: a collection of expectations written in Data Quality Definition Language, or DQDL. Add an Evaluate Data Quality transform to an ETL job at the point where those expectations should hold. Checks before cleanup measure incoming quality; checks after cleanup validate the prepared data before publication.

A uniqueness rule says that no two records may share the same key within a defined dataset. Choose the key from what one row represents, not from which columns happen to look unique today. A generated row identifier can be unique even when two rows describe the same business event.

A reference check asks whether a child record points to an existing record in a parent dataset. For example, an order points to a customer. In a data lake, storing files and declaring schemas does not enforce this relationship; a validation job must check it.

Referential integrity asks whether a reference resolves to a valid target. For example, every order's customer_id should match a customer in the Customers dataset. A reference to a missing customer is an orphan reference.

Consistency diagnostics investigates whether values agree where they should: between related fields, between datasets describing the same thing, or before and after processing. Agreement does not prove accuracy; two datasets can repeat the same wrong value. The goal is to find a disagreement, narrow its scope, and gather evidence about its cause.

Random sampling selects records by chance from a defined population: the complete set of records you want to learn about. A sample is the subset selected for inspection. Sampling lets you investigate data quality without checking every record.

Stratified sampling divides a population into groups called strata, then randomly selects records within every group. Start by defining the population: for example, all transactions ingested during one completed day. Keep that population fixed while counting and sampling.

Unstratified random sampling selects records from the population as one pool, without reserving places for particular groups. In the equal-probability case, every eligible record has the same chance of inclusion. Group counts in the sample are outcomes of the draw, not fixed targets.

A target population is the set of people, events, or records your conclusions should describe. Define the unit and scope before sampling: for example, all support conversations during a specified month, across every supported language and channel. A sample represents that intended population, not simply whatever data is easiest to retrieve.

Repartitioning changes how records are distributed across processing partitions. A partition is a unit of work, not necessarily a server. The goal is to balance the work needed by the next operation, rather than simply make every partition contain the same number of rows.

A distributed join must bring matching rows together. In a shuffle join, inputs can be repartitioned by the join key so matching rows reach the same worker. A hot key is a value that appears much more often than others. Sending all its rows to one partition can leave one task busy while others finish: this is join-key shuffle skew.

A hot key has far more records than other keys. In a distributed aggregation, grouping all those records together can leave one task doing much more work than the others. Key salting creates several temporary groups for that one business key.

In a distributed join, a hot key has enough records to overload the task handling it. Coordinated key salting gives that key several bucket identifiers so its join work can reach multiple tasks. Consider an inner join between many events and a smaller customer table, using customer identifier as the business key.

A hot key is a key shared by a disproportionately large amount of work. Suppose most event records belong to customer C123. With hash partitioning by customer, every occurrence of C123 goes to the same partition because the partitioning rule receives the same value each time.

An AWS security group filters traffic for its associated network interfaces using allow rules. Inbound rules permit traffic entering an interface; outbound rules permit traffic leaving it. Rules from all attached security groups are combined: a new connection needs a matching allow, and security groups have no explicit deny rules.

IAM users are identities created in an AWS account. A user can have a console password or access keys, but creating a user alone grants no access. Permissions come from policies. For workloads and temporary access, prefer roles over distributing long-lived user access keys.

AWS Identity and Access Management (IAM) defines identities and access rules. You use it to create users, organize IAM users into groups, create roles, and manage policies that govern access to AWS resources.

Credential rotation replaces a password at the database, publishes the replacement through a stored secret, and moves application clients onto it. Generating or storing a new password alone does not change what the database accepts.

Least privilege means granting only the operations needed for a task, on only the resources needed, under the required circumstances. Write a custom policy when a managed policy includes unnecessary permissions or cannot express your restrictions. Start with the application's actual API operations, not a broad service permission such as s3:*.

An IAM role is an identity with permissions, not a permanent access key to distribute. An authorized caller assumes the role, and AWS Security Token Service (AWS STS) issues credentials for a role session. Those credentials contain an access key ID, secret access key, and session token, with an expiration time. Requests need the token as well as the key pair.

Workload identity gives software an identity for AWS requests. An IAM role has no permanent access keys of its own. Instead, an authorized principal assumes it and receives temporary credentials for a role session. Authentication establishes the caller's identity; authorization decides whether its request is allowed.

AWS Identity and Access Management (IAM) defines permissions for identities within an AWS account. An identity-based policy attached to a role can grant permission to perform an action, such as starting a backup copy.

An identity-based policy is attached to an IAM user, group, or role. It describes what that identity can do to specified resources. For a data-processing application using a role, put the application's permissions on that role—not on the compute service running it.

An interface endpoint provides private connectivity to a service through AWS PrivateLink. For AWS services that support endpoint policies, an endpoint policy restricts which principals, actions, and resources can be used through that endpoint. Check the service’s endpoint-policy support before relying on this control.

Explicit deny precedence means that an applicable deny statement blocks a request even when another policy allows it. In AWS authorization, a matching deny in a role policy, resource policy, or applicable VPC endpoint policy cannot be overcome by an allow in another layer. Policies do not vote: several allows cannot outweigh one matching deny.

Effective permissions are the result for a particular caller, action, resource, and request context—not a list copied from the role's policies. Here, an application uses an assumed IAM role to request s3:GetObject through an S3 gateway VPC endpoint and an S3 access point.

Reading an encrypted resource can require two separate permissions: permission to access the resource and permission to use its encryption key. For an Amazon S3 object encrypted with AWS KMS, a reader needs s3:GetObject for the object and kms:Decrypt for the KMS key.

The shared responsibility model separates security of the cloud from security in the cloud. AWS protects facilities, physical hardware, networking infrastructure, and the virtualization layer. Customers remain responsible for their data, identities, permissions, and the workload settings they control.

An Amazon EC2 instance gives you a virtual machine whose guest operating system you administer. AWS operates the physical facilities, hardware, and virtualization infrastructure underneath it. You do not maintain the physical server, but you do maintain the operating system running inside your instance.

AWS IAM Identity Center provides workforce sign-in for a SageMaker Unified Studio domain configured to use it. It gives the application a way to recognize authenticated people and use centrally managed users and groups, rather than maintaining separate application passwords.

When people access SageMaker Unified Studio through AWS IAM Identity Center, Identity Center provides the workforce sign-in path. It connects the application to workforce identities, including identities from a connected identity provider. This answers who is signing in? It does not, by itself, authorize every action that person wants to perform.

A database password or external API token is an application secret. Keep its value out of source code, container images, and deployment templates. Store the value centrally, and configure the application with its secret identifier or parameter name instead.

Parameter Store, a capability of AWS Systems Manager, stores named configuration values. Give a database password a name such as /prod/orders/db-password, and store its value as a SecureString. The application can keep the parameter name in its configuration instead of embedding the password in source code.

Role-Based Access Control (RBAC) bundles permissions around a job, such as reporting analyst or data loader. A permission allows an action on a defined resource. A role collects the permissions needed for a responsibility, so administrators do not have to recreate the same grants for every person.

In Amazon Redshift, a database user is an identity under which SQL runs. Being able to authenticate does not automatically give that user permission to read every table. IAM permissions for AWS operations and Redshift database privileges serve different purposes: permission to manage a warehouse is not a SQL SELECT grant.

Attribute-Based Access Control (ABAC) authorizes requests by comparing attributes with policy rules. Authentication establishes who is making the request; authorization determines whether that requester may perform the requested action on the target resource.

A resource tag is metadata such as Project=Atlas. It does not grant access by itself. In attribute-based access control (ABAC), a permission policy uses attributes in conditions to decide whether a permission applies. This can let one policy cover multiple resources with the same business attributes rather than listing each resource separately.

Role-Based Access Control (RBAC) organizes permissions around approved responsibilities, such as analyst or application operator. A role bundles allowed actions and their resource scope; assigning someone that role gives them those permissions. RBAC does not mean broad access: a role can be limited to particular datasets or environments.

Least privilege means allowing the work an identity must perform without adding unrelated capabilities. Start with a concrete requirement: “The invoice reader lists and downloads current objects under invoices/ in billing-data-example, using HTTPS.” This identifies required operations, a data boundary, and a request constraint. It does not require uploading, deleting, or changing access policies.

Privilege escalation means gaining capabilities beyond an identity's approved authorization. In AWS, it can happen through valid API calls. A policy that blocks direct access to sensitive data is not enough if the identity can change its permissions or control something that already has that access.

Data masking limits disclosure of selected values while leaving useful context available. A support report might show a phone number as ***-***-0142 alongside an error code. The reader can troubleshoot without seeing the full number, but the visible digits still disclose some information.

Start with the disclosure policy: who may receive which values, for what purpose, and through which paths? Then identify the minimum information the task needs. A support agent may need a card’s last four digits, while a development team may need no real card digits at all. A masking technique is suitable only if its output meets the policy.

Pseudonymization replaces a direct identifier with a substitute while protecting the additional information that can connect it to a person. Replacing an email address with a stable customer token lets analysts join that customer's transactions without receiving the email address.

Cryptographic hashing maps input bytes to a fixed-length digest. A data pipeline can replace an email address with its SHA-256 digest before making an analytics dataset available. That digest acts as a pseudonym: a substitute identifier that lets analysts work without seeing the original field.

In envelope encryption, a data key encrypts the application’s data. A separate KMS key encrypts that data key. Here, an application on Amazon EC2 performs encryption locally and stores encrypted packages in Amazon S3; this is application-side encryption, not S3 server-side encryption.

AWS KMS manages cryptographic keys and performs operations with them. This lesson uses symmetric encryption KMS keys: the same protected key material supports encryption and decryption, but applications do not receive that KMS key material.

Envelope encryption separates encrypting data from protecting the key that encrypts it. A data encryption key, or DEK, encrypts the actual payload locally in the application. A separate key-encryption key, or wrapping key, protects that data key.

An AWS KMS key is a logical key resource, not just a single set of cryptographic bits. Its key material performs the cryptographic work. Rotation introduces fresh material within the same key resource. Think of versions A and B as teaching labels for successive sets of material, not identifiers your application must select.

Cross-account key use lets a principal in one AWS account request cryptographic operations with a KMS key owned by another account. The owner shares permission, not the key's cryptographic material. Use a customer managed key: its owner controls the key policy. You cannot edit an AWS managed key's policy to authorize another account.

For direct cross-account access, a workload keeps its existing identity while accessing another account's resource. Consider an AnalyticsReader role in Account A reading objects from a bucket in Account B. It does not assume a role in Account B. In this example, S3 Object Ownership is set to Bucket owner enforced, so Account B owns the objects and ACLs are disabled.

Encryption in transit protects data on a connection against interception and modification. Choose a secure protocol for each transfer: HTTPS is HTTP over TLS, FTPS is FTP protected by TLS, and SFTP uses SSH. Plain HTTP and plain FTP do not provide this protection. Encrypting stored files does not secure their transfer connections.

Transport Layer Security (TLS) protects application traffic against disclosure and undetected modification while it crosses a network. HTTPS is HTTP carried over TLS. During the TLS handshake, the endpoints agree on security parameters and establish keys before sending protected application data.

Client-side encryption means the application encrypts data before sending it to storage. The application can run on a laptop, on premises, or on cloud compute. What matters is where encryption happens, not where the application runs.

Encryption in transit protects data as it travels between connection endpoints. With TLS, the sender encrypts traffic and the receiving endpoint removes that transport encryption. If an application sends an ordinary readable payload over HTTPS, the receiving service can receive that payload as plaintext. This is expected behavior, not a failure of TLS.

Authentication identifies a requester. Authorization decides whether that requester may perform an action on a particular resource. Signing in, joining a project, or finding a dataset in a catalog does not by itself grant permission to query the data.

AWS Resource Access Manager (AWS RAM) lets an owning account share supported resources without copying them or transferring ownership. A resource share joins three things: the resources being shared, the recipient principals, and the RAM permissions that define allowed actions.

AWS Resource Access Manager (AWS RAM) establishes sharing relationships for supported resource types. An owner makes a resource available to selected recipients under the share's permissions. The resource stays in the owner's account; sharing does not copy it or transfer ownership.

Amazon Macie discovers sensitive data, including personally identifiable information (PII), in Amazon S3 objects. A sensitive data discovery job defines which S3 data to analyze and which detection rules to apply. It inspects object contents, rather than deciding sensitivity from object names alone.

Automated PII discovery finds content that resembles personally identifiable information. Amazon Macie analyzes supported data in Amazon S3 using managed data identifiers and custom data identifiers. A finding helps locate potentially sensitive content; it does not establish the organization's final sensitivity label.

Data residency describes where data is stored or processed geographically. A residency requirement identifies an allowed boundary, such as a country or a set of approved cloud Regions. That requirement can come from a contract, internal policy, or law.

Data residency requirements become usable engineering rules when they name the data in scope and its approved storage and processing locations. Confirm the interpretation with the policy owner or legal and compliance teams, then map the geographic boundary to an explicit list of approved AWS Regions and services. Do not assume that a broad label such as “Europe” satisfies a country-specific requirement.

AWS Organizations lets you attach a service control policy (SCP) to the organization root, an organizational unit (OU), or a member account. SCPs require the organization’s all features mode. A Region restriction attached to an OU applies to accounts in that OU, including accounts inside its nested OUs.

AWS Config records resource configurations as configuration items. Each item describes one resource at a point in time: its identity, recorded configuration settings, supported relationships to other resources, and metadata such as the capture time.

Turn jurisdictional requirements into an approved location list with legal and compliance teams before deploying. Identify where the dataset may be stored, processed, accessed, and recovered. Choose AWS Regions within that boundary, and verify each service's documented handling of customer data rather than assuming every service behaves identically.

Key custody describes who generates key material, which systems hold it, who authorizes its use, and who can recover or destroy it. Possession concerns where material is held; control concerns the authority to permit operations or change its lifecycle. Those responsibilities can belong to different parties.

A data governance framework defines who may decide how data is shared, which rules apply, and what evidence must be kept. A dataset being technically accessible does not make its use authorized. Sharing decisions must connect a named recipient and business purpose to the dataset’s sensitivity and obligations.

Data Mesh distributes responsibility for analytical data to business domains. Payments, for example, owns its settlements data product rather than handing every change or defect to a central data team. A data product includes an accountable owner, business definitions, discoverable metadata, supported access interfaces, and measurable quality and freshness expectations—not just files or tables.