# Backpressure

Backpressure is feedback from a constrained downstream component that causes upstream components to slow or pause production.

Cloud concept. Updated 2026-10-29. Page: https://blowstack.com/tools/cloud-concepts-hub/backpressure

> Backpressure keeps data producers from overwhelming consumers, processing stages, or storage systems when incoming work exceeds available capacity. It helps pipelines remain stable during traffic bursts, throttling, and slow transformations, but requires explicit policies for buffering, retries, and overload.

Practice exams covering Backpressure: [DEA-C01 AWS Certified Data Engineer - Associate](https://blowstack.com/practice-exams/aws-certified-data-engineer-associate), [DVA-C02 AWS Certified Developer - Associate](https://blowstack.com/practice-exams/aws-certified-developer-associate), [DOP-C02 AWS Certified DevOps Engineer - Professional](https://blowstack.com/practice-exams/aws-certified-devops-engineer-professional), [SAA-C03 AWS Certified Solutions Architect - Associate](https://blowstack.com/practice-exams/aws-certified-solutions-architect-associate).

## 6 key aspects of Backpressure

Effective backpressure combines downstream feedback, bounded work, safe retries, and bottleneck-aware recovery to maintain stability under overload.

1. **Downstream-driven feedback**: Use downstream readiness, throttling, queue occupancy, or processing latency to control upstream activity. A buffer alone is not backpressure unless its state influences admission or production.
2. **Bounded buffering and concurrency**: Limit queued work, active workers, and outstanding requests to prevent resource exhaustion. Define what happens when those limits are reached: pause, persist, reject, or intentionally discard according to the workload's requirements.
3. **Controlled retries**: Apply bounded exponential backoff with jitter to retryable failures and honor retry guidance when available. Retry only failed work when the API supports partial success, and make repeated processing safe.
4. **Bottleneck-aware scaling**: Identify whether capacity is constrained by partitions, compute, database connections, locks, or storage throughput before adding workers. Increasing upstream concurrency without increasing the constrained capacity can amplify failures.
5. **Lag and retention management**: Monitor backlog size and oldest-record age alongside throughput and errors. Ensure retention covers the expected disruption and recovery time, and verify that recovery throughput can exceed continuing arrivals.
6. **Explicit overload trade-offs**: Choose acceptable latency, resource consumption, and loss behavior for each workload. Critical business events may require durable buffering, while selected diagnostic logs may permit sampling or dropping under a documented policy.

## Cloud services for Backpressure

Kinesis Data Streams or SQS can hold incoming work while consumers regulate processing to protect downstream systems. Managed Service for Apache Flink can propagate pressure within a streaming application, while writers to DynamoDB or RDS should bound concurrency and respond appropriately to throttling or saturation. CloudWatch provides metrics and alarms for detecting lag and capacity constraints, and log shippers need their own overload controls. These services do not create an automatic end-to-end feedback loop: applications must connect capacity signals to admission, polling, and retry decisions, with retention and overflow policies covering prolonged overload.

- **[Amazon Kinesis Data Streams](https://blowstack.com/tools/cloud-services-hub/amazon-kinesis-data-streams)**: Streams decouple producers from consumers, allowing consumer lag to grow when processing slows. Monitor write throttling and iterator age, address partition-key distribution and capacity, and remember that consumer backpressure does not automatically reduce producer writes.
- **[Amazon Managed Service for Apache Flink](https://blowstack.com/tools/cloud-services-hub/amazon-managed-service-for-apache-flink)**: Flink can propagate backpressure upstream through its processing graph when downstream operators cannot keep up. Investigate slow sinks, skewed operators, state access, and effective parallelism before scaling the application.
- **[Amazon DynamoDB](https://blowstack.com/tools/cloud-services-hub/amazon-dynamodb)**: Throttling and unprocessed batch items provide signals for reducing ingestion pressure. Use backoff with jitter, inspect hot keys and capacity limits, and avoid treating additional client concurrency as a solution to insufficient write capacity.
- **[Amazon RDS](https://blowstack.com/tools/cloud-services-hub/amazon-rds)**: Applications protect relational databases by limiting concurrent connections and transactions and adjusting ingestion batches. Connection exhaustion, lock waits, and storage latency should trigger investigation and upstream pacing rather than uncontrolled retries.
- **[Amazon CloudWatch](https://blowstack.com/tools/cloud-services-hub/amazon-cloudwatch)**: CloudWatch metrics and alarms help detect throttling, rising backlog, and resource saturation. CloudWatch Logs ingestion clients need bounded buffering and retry behavior when requests are throttled or delivery is interrupted.

## Backpressure exam questions

Real practice exam questions on Backpressure, with the correct answer and its explanation.

### Question 1 (DEA-C01)

**Question:**

A company runs an AWS Glue streaming ETL job that reads events from Amazon Kinesis Data Streams. A custom transformation submits asynchronous requests to an external enrichment API. The API's available capacity occasionally decreases for several minutes, and throttled requests must be retried. During these periods, the transformation continues submitting requests, its in-memory pending-request queues grow, and Glue executors eventually fail with out-of-memory errors. CPU utilization remains low, and records are evenly distributed across executors.

The API's capacity cannot be increased. The Kinesis stream has sufficient retention to accommodate the temporary backlog, and the job can catch up after API capacity recovers. Every event must be processed. Which change should the data engineer make to keep the job stable?

**Correct answer:** Bound pending queues and in-flight requests, block further submission when those limits are reached, and retry throttled requests with exponential backoff and jitter. Make each micro-batch wait for its enrichment work to complete.

**Explanation:** This makes downstream capacity constrain upstream processing. Bounded queues and request concurrency prevent unbounded memory growth, while waiting for completion allows slower processing to delay subsequent micro-batches. Unread events remain durably buffered in Kinesis until capacity recovers. Backoff and jitter reduce retry pressure on the API.

### Question 2 (DOP-C02)

**Question:**

A company uses an Amazon CloudWatch Logs subscription to send application logs to a processor that writes to an external security platform. The log groups use the Standard log class and retain events for 30 days. During maintenance, the security platform can be unavailable for up to 48 hours. Processing must normally occur within seconds, and every event must remain available for processing after an outage. Duplicate processing is acceptable.

The processor supports stream consumption with persistent checkpoints. After the platform recovers, sufficient processing capacity is available to clear the backlog within 12 hours while handling new events. The company does not want to implement a separate log retrieval or archive replay workflow.

Which ingestion design best meets these requirements?

**Correct answer:** Send logs through a subscription to Amazon Kinesis Data Streams with 72-hour retention and sufficient ingestion capacity. Keep accepting subscription records during outages. Pause processing when the platform is unavailable, and advance persistent checkpoints only after successful processing.

**Explanation:** The stream durably buffers incoming events independently of the unavailable platform. Its 72-hour retention exceeds the maximum 48-hour outage plus the 12-hour catch-up period. Resuming from successful checkpoints avoids skipping unprocessed records, although retries can produce acceptable duplicates. Keeping stream ingestion available avoids relying on the shorter CloudWatch Logs subscription retry window.

### Question 3 (DVA-C02)

**Question:**

A document-indexing application uses a single worker process to consume messages from an Amazon SQS standard queue. The worker continuously receives messages and starts an asynchronous indexing request for each message without waiting for earlier requests to finish. It deletes messages only after successful indexing, extends visibility while processing, and retries throttled requests with exponential backoff and jitter.

During temporary slowdowns, the indexing endpoint returns HTTP 429 responses. The worker's pending requests and memory usage continue to grow even though retries are delayed. Messages can remain in the queue during these slowdowns, and the queue's retention period is sufficient.

Which change should a developer make to keep the worker's resource usage bounded and make its consumption respond to downstream capacity?

**Correct answer:** Use a bounded pool of in-flight indexing operations. Receive messages only when slots are available, fetch no more messages than available slots, and reduce the pool's concurrency when the endpoint throttles requests.

**Explanation:** The bounded pool limits outstanding work, and pausing receives when the pool is full leaves excess work durably buffered in SQS. Reducing concurrency in response to throttling propagates downstream capacity constraints to message consumption. Existing visibility extension, retry, and deletion behavior preserves unfinished work.

### Question 4 (SAA-C03)

**Question:**

A company sends sensor events from factory gateways to Amazon Kinesis Data Streams. A consumer transforms the events by calling a licensed processing system whose capacity cannot be increased. During normal operation, the processing system handles events faster than they arrive. Occasionally, its available capacity decreases for several hours, causing processing delays to grow even though Kinesis reports no read or write throttling.

The gateways can durably retain unsent events for 7 days. Events already submitted to Kinesis must be processed within 30 minutes, but events can remain at the gateways longer during processing slowdowns. The consumer publishes the age of its oldest unprocessed event as a custom Amazon CloudWatch metric.

The company wants ingestion to adapt automatically to processing slowdowns and return to its normal rate after recovery. Which solution meets these requirements?

**Correct answer:** Use the event-age metric to control gateway submission rates. Reduce or pause submissions before processing delay approaches 30 minutes, retain unsent events at the gateways, and gradually restore submissions as the delay decreases.

**Explanation:** This establishes feedback from downstream processing to upstream ingestion. The controller reduces incoming work when processing falls behind and restores throughput when capacity returns. Durable gateway storage holds deferred events outside the stream, protecting the processing deadline for events already accepted.

## Practice Backpressure in a full exam

Questions on Backpressure in each practice exam, with every answer option and an explanation for each:

- [DEA-C01](https://blowstack.com/practice-exams/aws-certified-data-engineer-associate): 5 questions
- [DOP-C02](https://blowstack.com/practice-exams/aws-certified-devops-engineer-professional): 2 questions
- [SAA-C03](https://blowstack.com/practice-exams/aws-certified-solutions-architect-associate): 2 questions
- [DVA-C02](https://blowstack.com/practice-exams/aws-certified-developer-associate): 1 question

## What is Backpressure

Backpressure is a feedback mechanism that connects the rate of producing work to the capacity available to process it. When a downstream component becomes saturated, upstream components reduce concurrency, delay requests, or pause reading. The feedback can be explicit, such as a throttling response, or indirect, such as a full bounded queue, increasing processing latency, or blocked output writes. The defining behavior is that downstream constraints influence upstream activity.

Consider an ingestion pipeline receiving 10,000 records per second while its database can sustainably accept only 6,000. Without intervention, the backlog grows by approximately 4,000 records every second. A buffer can absorb a temporary burst, but it cannot resolve this sustained mismatch. Backpressure reduces admission or consumption to a sustainable rate while engineers increase downstream capacity, improve processing efficiency, or arrange durable storage for deferred work. If the source cannot slow down, the architecture must explicitly handle buffering, rejection, or intentional data loss.

In cloud applications, enforce bounded queues and limits on in-flight requests rather than creating unlimited tasks. A consumer can stop fetching messages when its worker pool is full and resume as capacity becomes available. A streaming framework can propagate pressure through connected operators when a sink blocks. However, pressure inside a consumer does not necessarily slow an independent producer writing to a durable stream. In that case, records accumulate in the stream, and retention becomes a recovery deadline.

Service throttling is a useful overload signal, but throttling and backpressure are not identical. A fixed request-rate limit is a policy; backpressure is the response that adjusts upstream behavior to downstream constraints. For DynamoDB ingestion, inspect throttling causes, partition-key distribution, and available capacity. Retry unprocessed batch items with bounded exponential backoff and jitter rather than repeatedly resubmitting the entire successful portion of a batch. More workers can worsen a hot partition instead of increasing useful throughput.

For Amazon RDS ingestion, increasing writer concurrency can exhaust connections or increase lock contention, CPU usage, and storage latency. Limit concurrent transactions, use appropriately sized batches, and investigate slow queries and indexes before increasing parallelism. Database saturation is not necessarily an API rate-limit problem. Similarly, Kinesis write throttling calls for examining stream capacity and partition-key distribution; consumer lag requires investigating read capacity, processing speed, and sink performance separately.

Transformation bottlenecks can arise from expensive parsing, skewed keys, large state, slow external lookups, or insufficient resources. A slow sink may make upstream operators appear stalled even when their own logic is efficient. Compare input and output rates, per-stage processing time, queue depth, oldest-record age, throttling, and resource utilization to find the actual constraint. Scale the constrained stage only when the source partitions, processing model, and destination capacity allow that scaling to help.

For real-time logs and machine-learning data ingestion, size durable buffers and retention for the expected outage and recovery period. A logging agent should use bounded buffering and an explicit overflow policy so that a remote logging outage does not exhaust application memory or disk. Monitor data freshness as well as throughput: a pipeline can remain operational while delivering logs or training data hours late.

Backpressure trades immediate throughput and latency for stability and controlled resource use. It does not guarantee lossless or exactly-once processing. Use idempotent writes or deduplication where retries can repeat work, limit retry attempts or elapsed time, and isolate permanently invalid records instead of retrying them indefinitely. After recovery, drain the backlog at a rate that leaves capacity for new arrivals rather than creating another overload cycle.

## Learn Backpressure in lessons

4 short lessons on Backpressure in 3 certification courses.

- DEA-C01: Backpressure: capacity-driven flow control (Premium), https://app.blowstack.com/lessons/backpressure-capacity-driven-flow-control-dea-c01
- DOP-C02: Backpressure: controlling flow in log ingestion pipelines (Premium), https://app.blowstack.com/lessons/backpressure-controlling-flow-in-log-ingestion-pipelines-dop-c02
- SAA-C03: Backpressure: controlling flow through streaming pipelines (Premium), https://app.blowstack.com/lessons/backpressure-controlling-flow-through-streaming-pipelines-saa-c03
- DEA-C01: Backpressure vs. rate limiting (Premium), https://app.blowstack.com/lessons/backpressure-vs-rate-limiting-dea-c01
