# Chaos Engineering

Chaos engineering is the practice of running controlled failure experiments to test how systems and operational teams respond to disruption.

Cloud concept. Updated 2026-10-29. Page: https://blowstack.com/tools/cloud-concepts-hub/chaos-engineering

> Chaos engineering uses explicit hypotheses, realistic faults, and measurable outcomes to uncover weaknesses before an unplanned incident exposes them. In cloud environments, it validates both technical resilience and incident response procedures, including detection, escalation, containment, recovery, and communication.

Practice exams covering Chaos Engineering: [DOP-C02 AWS Certified DevOps Engineer - Professional](https://blowstack.com/practice-exams/aws-certified-devops-engineer-professional), [SCS-C03 AWS Certified Security - Specialty](https://blowstack.com/practice-exams/aws-certified-security-specialty), [SAP-C03 AWS Certified Solutions Architect - Professional](https://blowstack.com/practice-exams/aws-certified-solutions-architect-professional).

## 6 key aspects of Chaos Engineering

Effective chaos engineering combines measurable hypotheses, limited failure scope, reliable observation, recovery safeguards, and repeated validation of systems and incident response procedures.

1. **Hypothesis-driven experiments**: Define the expected system and responder behavior before introducing a fault. Use measurable acceptance criteria, such as maximum error rate, alert delivery time, and recovery duration.
2. **Controlled blast radius**: Limit experiments to explicitly selected resources, workloads, and environments. Use least-privilege permissions, approved execution windows, and small initial scopes before increasing realism.
3. **Observable steady state**: Establish baseline customer-facing and infrastructure measurements so that deviations are detectable. Confirm that telemetry remains available during the failure being tested.
4. **Abort and recovery safeguards**: Define stop thresholds and provide an operator-controlled abort path. Prepare recovery actions separately because stopping fault injection does not guarantee that affected resources return to their original state.
5. **End-to-end incident response validation**: Test detection, paging, triage, escalation, containment, recovery, and communication rather than checking failover alone. Include evidence preservation and authorized security simulations when validating security incident procedures.
6. **Repeatable learning and remediation**: Capture experiment results, identify gaps, and assign corrective actions. Rerun the experiment after changes and periodically repeat it as infrastructure and runbooks evolve.

## Cloud services for Chaos Engineering

AWS Resilience Hub helps identify resilience gaps and recommends operational controls and experiments. AWS Fault Injection Service executes supported disruptions against deliberately scoped targets, while Amazon CloudWatch measures customer impact and supplies alarm-based stop conditions. AWS Systems Manager provides repeatable response and recovery automation that teams can exercise during testing. AWS CloudTrail supplies an audit record of relevant API actions for reconstructing the timeline and reviewing responder behavior. Together, these services support a cycle of assessment, controlled experimentation, observation, recovery, and remediation. They do not replace approvals, human coordination, or dedicated simulations of security threats that infrastructure faults do not represent.

- **[AWS Resilience Hub](https://blowstack.com/tools/cloud-services-hub/aws-resilience-hub)**: Assesses application resilience against a resilience policy containing recovery time and recovery point objectives. Its recommendations can include alarms, standard operating procedures, and AWS FIS experiments, helping teams select tests rather than replacing their execution or proving incident response readiness.
- **[Amazon CloudWatch](https://blowstack.com/tools/cloud-services-hub/amazon-cloudwatch)**: Provides metrics, logs, dashboards, and alarms for establishing baseline behavior and measuring experiment impact. Its alarms can trigger incident notifications and act as AWS FIS stop conditions when defined thresholds are breached.
- **[AWS Systems Manager](https://blowstack.com/tools/cloud-services-hub/aws-systems-manager)**: Automation runbooks can implement repeatable response and recovery procedures that experiments validate. Run Command can execute authorized commands on managed nodes for supported test scenarios, subject to appropriate targeting and access controls.
- **[AWS CloudTrail](https://blowstack.com/tools/cloud-services-hub/aws-cloudtrail)**: Records AWS API activity to help reconstruct experiment actions and responder activity. With appropriate event coverage and retention configured, its records support investigation, evidence review, and validation of security incident response procedures.

## Chaos Engineering exam questions

Real practice exam questions on Chaos Engineering, with the correct answer and its explanation.

### Question 1 (SAP-C03)

**Question:**

A financial services company processes payment instructions asynchronously. An API returns HTTP 202 after placing each instruction in a queue. Amazon EC2 workers then settle the payments with a downstream system. API availability does not indicate that payments have settled.

Under a controlled, constant load, the application normally settles 1,000 payments per minute. The business permits a temporary reduction of up to 5% in each 1-minute interval during a disruption, provided that the oldest unsettled payment is no more than 120 seconds old at each interval boundary.

A solutions architect is preparing a 15-minute AWS Fault Injection Service (AWS FIS) experiment that temporarily stops selected workers. The application publishes a CloudWatch count metric with a value of 1 for each confirmed settlement. An independent monitor publishes the age of the oldest unsettled payment at each minute boundary, including zero when no payments are pending. Both metrics are available before and throughout the experiment.

Which experiment hypothesis and measurement plan best validate the business requirements?

**Correct answer:** Hypothesize that the remaining workers will preserve acceptable settlement performance. Require the CloudWatch Sum of confirmed settlements to be at least 950 in every 1-minute period and each oldest-unsettled-payment age observation to be at or below 120 seconds. Treat the experiment as successful only if both conditions hold throughout the experiment.

**Explanation:** This hypothesis describes successful application behavior during the disruption using business outcomes. Summing the per-settlement count metric measures completed payments in each required interval, and the age metric independently validates the pending-work limit. A steady-state definition can allow bounded degradation: the permitted 5% throughput reduction does not itself represent failure.

### Question 2 (SCS-C03)

**Question:**

A company's incident response plan requires an evidence-export service to preserve every artifact listed in an incident manifest in a protected Amazon S3 bucket within 10 minutes, even if an export worker fails. The service runs on Amazon EC2 and uses a durable work queue. Each manifest includes the expected SHA-256 digest of every artifact.

In a representative nonproduction environment, an AWS Fault Injection Service (AWS FIS) experiment stopped an export worker during an active export. The experiment completed successfully, replacement capacity became healthy, and the queue eventually emptied. However, reviewers discovered that some manifest entries had no corresponding object in the destination bucket. Developers have changed the worker's retry logic.

Which procedure BEST validates whether the updated service meets the incident response requirement?

**Correct answer:** Repeat the approved worker-stop experiment during an active export of synthetic artifacts. Measure from export initiation until every manifest entry has a destination object with the expected digest, enforce the exercise's safety limits, and compare the result with the 10-minute objective.

**Explanation:** This tests a measurable hypothesis using a realistic fault and evaluates the required incident response outcome. Manifest reconciliation and digest verification establish completeness and integrity, while elapsed time establishes whether the preservation deadline was met. Repeating the fault after the retry change demonstrates whether the identified weakness was corrected.

### Question 3 (DOP-C02)

**Question:**

A company runs a stateless customer portal in two AWS Regions. Amazon Route 53 failover records direct users to the primary Region unless its HTTPS health check fails. Both Regions serve the same immutable dataset. The company requires new user sessions to recover within 120 seconds after the primary application becomes unavailable, and the on-call team must receive an alert within 60 seconds.

Previous exercises changed the DNS records manually and confirmed that the secondary application worked. However, an incident later revealed that the primary endpoint's health check could remain healthy while customer requests failed.

A DevOps engineer must design a repeatable experiment in a production-equivalent test environment with a separate DNS hostname. An approved AWS Fault Injection Service (AWS FIS) action can stop the test environment's primary application instances. The engineer must limit the experiment to those instances, preserve the normal failure-detection and recovery paths, and stop further fault injection if an independent safety alarm enters ALARM state. Stopping the experiment does not restart instances that have already stopped.

Which experiment design best meets these requirements?

**Correct answer:** Target the tagged primary test instances with AWS FIS and configure the safety alarm as a stop condition. Send continuous requests through the test hostname using representative DNS caching behavior. Measure sustained application recovery and alert delivery from fault onset, retain failed health-check detection as an experiment failure, and run explicit cleanup to restart stopped instances.

**Explanation:** This experiment introduces an actual application failure without bypassing the normal health-check, DNS failover, or alerting paths. Client-level measurements include detection and DNS caching delays, so they can validate the 120-second recovery requirement. Measuring alert delivery validates the operational response. Explicit targeting limits the blast radius, the alarm stops further injection, and cleanup handles resources that stopping the experiment does not restore.

## Practice Chaos Engineering in a full exam

Questions on Chaos Engineering in each practice exam, with every answer option and an explanation for each:

- [SCS-C03](https://blowstack.com/practice-exams/aws-certified-security-specialty): 9 questions
- [SAP-C03](https://blowstack.com/practice-exams/aws-certified-solutions-architect-professional): 7 questions
- [DOP-C02](https://blowstack.com/practice-exams/aws-certified-devops-engineer-professional): 3 questions

## What is Chaos Engineering

Chaos engineering treats resilience as something to demonstrate through experiments rather than assume from an architecture diagram. Engineers define normal system behavior, predict what should happen during a specific failure, introduce that failure under controlled conditions, and compare the observed response with the prediction. The objective is to discover weaknesses and improve the system, not simply to cause an outage.

A useful experiment starts with a measurable hypothesis. For example, if one application instance becomes unavailable, healthy instances should continue serving requests, the error rate should remain below an agreed threshold, and the on-call engineer should receive an alert within two minutes. Measurements should include customer-facing outcomes such as request success and latency, not only infrastructure signals such as CPU utilization. The experiment should also define its duration, target resources, responsible operators, abort thresholds, and recovery procedures.

Cloud infrastructure makes many experiments repeatable through APIs and automation. Teams can interrupt selected compute capacity, introduce supported network disruptions, or stress resources to test behavior under degraded conditions. Start with a representative nonproduction environment, then consider carefully scoped production experiments when the remaining risk is understood and approved. Restrict targets through explicit resource selection, use least-privilege execution roles, and limit the blast radius so that an experiment cannot unexpectedly affect unrelated workloads or accounts.

For incident response validation, technical failover is only part of the outcome. An experiment should establish whether monitoring detects the symptom, whether the correct team is paged, whether responders can access required tools, and whether the runbook leads to an effective recovery. Record detection and recovery times, escalation delays, communication quality, and any manual steps that were missing or ambiguous. A simulated loss of an application dependency, for example, can expose an alert routed to the wrong team or a recovery procedure that depends on the failed component.

Security response exercises require additional care. Infrastructure fault injection does not automatically simulate credential theft, data exfiltration, or attacker behavior. To validate a security incident response plan, use authorized simulations and test resources to exercise detection, evidence preservation, containment, and restoration. For example, a controlled scenario involving a designated test identity can assess whether responders identify relevant audit events and revoke access without disrupting unrelated identities. Tabletop exercises complement technical experiments by testing decisions, responsibilities, and communications without modifying live infrastructure.

Safety controls are essential, but they have limits. Monitoring alarms can provide automatic stop conditions when customer impact exceeds an approved threshold. Stopping an experiment prevents further experiment actions where supported; it does not necessarily reverse changes already made. A terminated instance is not restored merely because the experiment stops. Recovery may require replacement capacity, configuration restoration, or a separate automation runbook. Verify those recovery paths before executing the experiment.

Chaos engineering involves operational risk, engineering effort, and potentially additional infrastructure cost. Nonproduction testing is safer but may miss production-only dependencies, traffic patterns, or access constraints. Production testing provides more realistic evidence but requires stronger safeguards and coordination. Common misconceptions are that chaos engineering means random destruction, that passing one experiment proves complete resilience, or that a resilience assessment replaces live testing. After each experiment, document findings, assign remediation owners, update the incident response plan, and repeat the test to demonstrate that the changes work.

## Learn Chaos Engineering in lessons

4 short lessons on Chaos Engineering in 3 certification courses.

- SAP-C03: Chaos Engineering: hypothesis-driven failure experiments (Premium), https://app.blowstack.com/lessons/chaos-engineering-hypothesis-driven-failure-experiments-sap-c03
- SCS-C03: Tabletop Exercises vs Chaos Engineering (Premium), https://app.blowstack.com/lessons/tabletop-exercises-vs-chaos-engineering-scs-c03
- DOP-C02: Chaos Engineering: controlled failover experiments (Premium), https://app.blowstack.com/lessons/chaos-engineering-controlled-failover-experiments-dop-c02
- SCS-C03: Chaos Engineering: hypothesis-driven fault experiments (Premium), https://app.blowstack.com/lessons/chaos-engineering-hypothesis-driven-fault-experiments-scs-c03
