# Exception Handling

Exception handling is the controlled detection and management of runtime failures so a program can recover, release resources, or terminate with useful diagnostic information.

Cloud concept. Updated 2026-10-29. Page: https://blowstack.com/tools/cloud-concepts-hub/exception-handling

> Data pipelines encounter malformed records, unavailable dependencies, permission errors, and unexpected program behavior. Exception handling separates failure paths from normal processing and makes recovery decisions explicit, helping engineers protect data integrity and operate reliable ingestion and transformation workloads.

Practice exams covering Exception Handling: [DEA-C01 AWS Certified Data Engineer - Associate](https://blowstack.com/practice-exams/aws-certified-data-engineer-associate), [DVA-C02 AWS Certified Developer - Associate](https://blowstack.com/practice-exams/aws-certified-developer-associate), [DOP-C02 AWS Certified DevOps Engineer - Professional](https://blowstack.com/practice-exams/aws-certified-devops-engineer-professional).

## 6 key aspects of Exception Handling

Effective exception handling combines precise failure classification, scoped handlers, safe recovery, data-integrity controls, cleanup, and observable failure paths.

1. **Explicit failure classification**: Separate transient dependency failures, invalid input, and unexpected programming errors. This classification determines whether to retry, quarantine a record, or terminate processing.
2. **Scoped, specific handlers**: Catch expected exception types close to the operation that can fail. Let unexpected failures propagate after adding useful context rather than masking defects with broad catch-all handlers.
3. **Bounded and safe recovery**: Use timeouts, retry limits, exponential backoff, and jitter for recoverable failures. Make repeated operations idempotent and coordinate retry policies across SDKs, application code, and orchestrators.
4. **Data integrity during partial failure**: Use transactions, deduplication, and carefully managed checkpoints to prevent missing or duplicate output. Quarantine rejected records with sufficient metadata for investigation and replay.
5. **Resource cleanup and propagation**: Release files, connections, and other resources with language-supported cleanup constructs. Preserve exception causes and ensure unrecoverable errors produce a failure status visible to the caller.
6. **Observable and tested failure paths**: Emit structured diagnostics and metrics for failures, retries, and rejected records without leaking sensitive data. Test recovery behavior under malformed input, dependency failures, and ambiguous write outcomes.

## Cloud services for Exception Handling

A data pipeline can use Step Functions to coordinate Glue jobs or Lambda functions, with EMR running larger distributed transformations. Application code handles expected parsing and validation failures, while workflow policies manage selected task-level retries and fallback paths. Each layer needs a clear retry budget so application, SDK, framework, and orchestration retries do not multiply unexpectedly. CloudWatch provides logs, metrics, and alarms for diagnosing failures and detecting incomplete processing. Across these services, idempotent writes, protected quarantine storage, and explicit checkpoint rules make recovery safer; no orchestration or monitoring service automatically guarantees correct output after partial failure.

- **[AWS Glue](https://blowstack.com/tools/cloud-services-hub/aws-glue)**: Glue jobs run ingestion and transformation code in which engineers implement record validation, exception handling, and rejection paths. Configured job retries can rerun failed work, so output writes and checkpoint behavior must tolerate repeated execution.
- **[AWS Step Functions](https://blowstack.com/tools/cloud-services-hub/aws-step-functions)**: Step Functions supports Retry and Catch policies on supported workflow states to distinguish retryable errors from failures that require another execution path. These orchestration policies complement application handlers but cannot catch every possible failure, including certain terminal errors.
- **[AWS Lambda](https://blowstack.com/tools/cloud-services-hub/aws-lambda)**: Lambda functions use runtime exception mechanisms to report processing failures. Retry and failed-event routing behavior depends on the invocation source; supported event source mappings can use partial batch responses to avoid reprocessing successful records when configured appropriately.
- **[Amazon EMR](https://blowstack.com/tools/cloud-services-hub/amazon-emr)**: Spark applications on EMR combine language-level handlers with distributed task-failure and retry behavior. Engineers must account for lazy evaluation and prevent repeated task execution from duplicating external side effects.
- **[Amazon CloudWatch](https://blowstack.com/tools/cloud-services-hub/amazon-cloudwatch)**: CloudWatch collects application logs and metrics that make exception paths observable. Alarms can highlight elevated error rates, exhausted recovery attempts, or growing rejection counts, but monitoring does not replace application recovery logic.

## Exception Handling exam questions

Real practice exam questions on Exception Handling, with the correct answer and its explanation.

### Question 1 (DEA-C01)

**Question:**

An AWS Glue Python shell job processes payment records sequentially. For each record, parse_amount(record) returns a validated amount or raises ValueError if the source amount is malformed. The job must save malformed records to a quarantine destination and continue processing subsequent records.

The write_payment(record, amount) function can also raise ValueError, but this indicates an output validation defect that must fail the job. Any failure when writing payments or quarantined records must propagate to AWS Glue. An output client must be closed whether processing succeeds or fails.

Which exception-handling structure meets these requirements?

**Correct answer:** Use an outer try/finally to close the output client. Inside the loop, place only parse_amount in an inner try block. Catch ValueError to quarantine the record, and invoke write_payment in the inner try statement's else block. Do not catch exceptions from either write operation.

**Explanation:** The narrow handler isolates malformed source amounts without intercepting output defects. Exceptions raised in the except or else blocks are not caught by that same try statement's handler, so quarantine and payment write failures propagate. The outer finally closes the client on both successful and failed runs.

### Question 2 (DOP-C02)

**Question:**

A company uses an Amazon Data Firehose delivery stream to deliver operational events to Amazon S3 for incident analysis. An AWS Lambda transformation function validates and normalizes each JSON event. Firehose data format conversion is disabled.

Each invocation can contain valid incident events, valid heartbeat events that must be discarded, and malformed events. The company requires valid incident events to reach the primary S3 prefix and malformed events to be retained in a separate S3 error prefix for investigation. Heartbeat events must not be delivered to either prefix. A malformed event must not cause successfully transformed events in the same invocation to be retried. The team wants to use Firehose's managed processing and error-delivery capabilities rather than implement separate S3 writes.

Which implementation meets these requirements?

**Correct answer:** Return Ok with normalized data for valid incident events, Dropped for heartbeat events, and ProcessingFailed for malformed events. Preserve the input recordId in every response entry, include the required base64-encoded data, and configure the Firehose S3 error output prefix.

**Explanation:** Firehose supports per-record transformation results. Ok records continue to the destination, Dropped records are intentionally excluded, and ProcessingFailed records are treated as unsuccessfully processed and delivered to S3 error storage. Returning a valid response for every input record isolates malformed records without failing the invocation or retrying successfully transformed records.

### Question 3 (DVA-C02)

**Question:**

A developer is troubleshooting a Node.js AWS Lambda function. The saveRecord function returns a Promise that resolves when a record is durably stored and rejects if the write fails. The handler contains the following code:

export const handler = async (event) => {
  event.records.forEach(async (record) => {
    await saveRecord(record);
  });
  return { processed: event.records.length };
};

The handler sometimes returns before the writes finish, and write rejections are not propagated through the handler's returned Promise. The function must return a successful result only after every write succeeds. Any rejected write must cause the invocation to fail. Which change meets these requirements?

**Correct answer:** Replace the forEach statement with await Promise.all(event.records.map(record => saveRecord(record))), keeping the return statement after it.

**Explanation:** map collects the write Promises, and Promise.all resolves only when every write succeeds. Awaiting it prevents an early success response. If a write rejects, the rejection propagates out of the async handler and causes the invocation to fail.

## Practice Exception Handling in a full exam

Questions on Exception Handling in each practice exam, with every answer option and an explanation for each:

- [DVA-C02](https://blowstack.com/practice-exams/aws-certified-developer-associate): 6 questions
- [DOP-C02](https://blowstack.com/practice-exams/aws-certified-devops-engineer-professional): 2 questions
- [DEA-C01](https://blowstack.com/practice-exams/aws-certified-data-engineer-associate): 1 question

## What is Exception Handling

Exception handling defines what a program does when an operation cannot complete normally. An exception interrupts ordinary execution and propagates until an appropriate handler processes it or the runtime terminates the affected operation. In data engineering, handlers should distinguish between failures that can be recovered automatically, records that require isolation, and conditions that should stop a pipeline. Catching an exception is not the same as correcting its cause.

Programming languages expose different mechanisms. Python uses try, except, else, and finally; Java uses try, catch, and finally, with try-with-resources for supported resources. Scala supports try and catch as well as explicit failure representations such as Try. R provides tryCatch and on.exit. SQL exception syntax depends on the database and procedural language; transaction rollback and statement error behavior must be checked for the engine being used. Bash generally communicates failure through exit status rather than language exceptions, so scripts must inspect command results and configure pipeline failure behavior deliberately. PowerShell uses try, catch, and finally, but catch handles terminating errors; many cmdlet errors require ErrorAction Stop to enter that path.

Place handlers around meaningful failure boundaries rather than wrapping an entire application in a catch-all block. For example, a Python ingestion process can catch a ValueError while parsing a numeric field, attach the source object and record identifier, and send the rejected record to a restricted quarantine location. A missing required column across an entire dataset may instead indicate an incompatible schema and justify failing the run. Silently substituting zero for every parsing failure can produce technically successful but financially incorrect output.

Cloud dependencies introduce transient failures such as throttling, temporary network interruptions, and service unavailability. Retry only appropriate failures, using bounded attempts, exponential backoff, and jitter. Invalid credentials or malformed requests generally require correction rather than repetition. Configure connection and request timeouts, and consider SDK retry behavior before adding another retry layer: nested retries can multiply requests, latency, and cost.

Recovery must account for partial completion. If a database write succeeds but its acknowledgment is lost, retrying the operation can create duplicates. Use idempotency keys, unique constraints, transactional writes, or deterministic output identifiers where appropriate. Do not advance a checkpoint until the corresponding output is durably committed under the pipeline's delivery design. A rollback only protects work within its transaction; it does not automatically reverse an external API call or an object already written elsewhere.

Distributed frameworks add another boundary. Spark transformations are often evaluated lazily, so an error may surface during an action rather than when the transformation is defined. A driver-side handler must surround the operation that triggers execution, while record-level validation belongs in the distributed processing logic. Task retries can repeat side effects performed inside executor code, so external writes require particular care.

Always release resources through context managers, finally blocks, or equivalent mechanisms, while recognizing that abrupt process termination can prevent cleanup code from running. Preserve the original exception when rethrowing or adding context. Record the error category, execution identifier, affected input, and retry attempt without exposing credentials or sensitive record contents. Avoid logging and suppressing unexpected failures: doing so can make an orchestrator report success for incomplete output. Test malformed input, exhausted retries, partial writes, and cleanup failures, not just the successful path.

## Learn Exception Handling in lessons

2 short lessons on Exception Handling in 2 certification courses.

- DEA-C01: Exception Handling: failure boundaries in pipeline code (Premium), https://app.blowstack.com/lessons/exception-handling-failure-boundaries-in-pipeline-code-dea-c01
- DOP-C02: Exception Handling: failure boundaries in event processors (Premium), https://app.blowstack.com/lessons/exception-handling-failure-boundaries-in-event-processors-dop-c02
