# Distributed Tracing

Distributed tracing follows a request across multiple application components to show how they interact, where time is spent, and where failures occur.

Cloud concept. Updated 2026-10-29. Page: https://blowstack.com/tools/cloud-concepts-hub/distributed-tracing

> A distributed trace connects individual operations across services into a single view of a request's execution. It helps engineers troubleshoot cloud applications whose behavior cannot be understood from one server's logs or metrics alone.

Practice exams covering Distributed Tracing: [CLF-C02 AWS Certified Cloud Practitioner](https://blowstack.com/practice-exams/aws-certified-cloud-practitioner), [DVA-C02 AWS Certified Developer - Associate](https://blowstack.com/practice-exams/aws-certified-developer-associate), [DOP-C02 AWS Certified DevOps Engineer - Professional](https://blowstack.com/practice-exams/aws-certified-devops-engineer-professional), [AIP-C01 AWS Certified Generative AI Developer - Professional](https://blowstack.com/practice-exams/aws-certified-generative-ai-developer-professional), [MLA-C02 (beta) AWS Certified Machine Learning Engineer - Associate](https://blowstack.com/practice-exams/aws-certified-machine-learning-engineer-associate), [SAA-C03 AWS Certified Solutions Architect - Associate](https://blowstack.com/practice-exams/aws-certified-solutions-architect-associate), [SAP-C03 AWS Certified Solutions Architect - Professional](https://blowstack.com/practice-exams/aws-certified-solutions-architect-professional).

## 6 key aspects of Distributed Tracing

Useful distributed tracing combines correlated spans, reliable context propagation, actionable timing and error data, and deliberate controls for telemetry cost and sensitive information.

1. **Request-wide correlation**: A shared trace identifier connects work performed by multiple components for the same request. Engineers can move from an entry-point failure to related downstream operations without matching events only by timestamps.
2. **Spans and execution relationships**: Spans describe individual operations, their timing, and their relationships to other operations. Parent-child relationships and span links help represent synchronous calls, parallel work, and asynchronous processing.
3. **Instrumentation and context propagation**: Instrumentation creates spans, while context propagation carries tracing identifiers across network and messaging boundaries. Both are necessary for a useful cross-service view; collecting isolated spans does not automatically produce a complete trace.
4. **Latency and failure investigation**: Trace timelines and dependency views help locate slow calls, repeated retries, and errors along a request path. They guide investigation but should be combined with logs, metrics, and application knowledge before assigning a root cause.
5. **Sampling and data protection**: Sampling controls telemetry volume and cost at the expense of retaining every request. Attribute filtering, access controls, and retention policies reduce the risk of exposing sensitive application data.
6. **Correlation with other telemetry**: Trace and span identifiers can be included in application logs to connect detailed events to a request's execution. Metrics provide aggregate signals that help engineers decide which traces to investigate.

## Cloud services for Distributed Tracing

For an AWS application using an API Gateway REST API and Lambda, X-Ray tracing can connect the API entry point with function execution. Instrumentation in the application adds visibility into supported downstream calls and custom operations, while trace-context propagation preserves their relationships. X-Ray provides request-level analysis, and CloudWatch supplies complementary logs, metrics, and application observability views. Engineers can begin with an elevated error-rate metric, inspect representative traces, and use correlated logs to investigate the affected operation. Coverage depends on integration support, instrumentation, sampling, and permissions; enabling one service's tracing setting does not automatically trace every dependency.

- **[AWS X-Ray](https://blowstack.com/tools/cloud-services-hub/aws-x-ray)**: AWS X-Ray collects and analyzes application trace data to help engineers investigate request latency, errors, and service dependencies. Applications need supported integrations or instrumentation to provide meaningful trace coverage.
- **[Amazon CloudWatch](https://blowstack.com/tools/cloud-services-hub/amazon-cloudwatch)**: Amazon CloudWatch combines monitoring and application observability capabilities with logs and metrics. Its tracing-related views and Application Signals capabilities help engineers connect application health and service dependencies with request-level investigation.
- **[AWS Lambda](https://blowstack.com/tools/cloud-services-hub/aws-lambda)**: AWS Lambda supports integration with AWS X-Ray so function executions can participate in distributed traces. Tracing calls made inside function code may require additional instrumentation beyond enabling tracing for the function.
- **[Amazon API Gateway](https://blowstack.com/tools/cloud-services-hub/amazon-api-gateway)**: Amazon API Gateway supports AWS X-Ray tracing for REST APIs, allowing the API entry point to participate in request traces. Support varies by API type, so tracing behavior should be checked rather than assumed for every API Gateway deployment.

## Distributed Tracing exam questions

Real practice exam questions on Distributed Tracing, with the correct answer and its explanation.

### Question 1 (MLA-C02 (beta))

**Question:**

A company operates a document-classification API with separate preprocessing, inference, and postprocessing services. OpenTelemetry instrumentation propagates context correctly across all services and sends spans through a collector tier to AWS X-Ray. Currently, every request is recorded and exported.

Only 1% of requests fail or take longer than 3 seconds. Engineers need to retain complete operation timelines for every such request, plus approximately 5% of the remaining requests, while reducing the volume of trace data stored in X-Ray. Requests finish within 20 seconds, and the collector tier has sufficient buffering capacity.

Which configuration best meets these requirements?

**Correct answer:** Continue recording all spans to the collector tier. Route every span for a given trace to the same collector, and apply tail sampling after allowing the trace to finish. Retain traces with errors or durations over 3 seconds, plus a probabilistic 5% sample of other traces.

**Explanation:** Tail sampling evaluates collected spans after request outcomes and durations are available. Routing each trace to one collector and allowing sufficient decision time lets the policy retain complete failed and slow traces while exporting only a small sample of normal traces to X-Ray.

### Question 2 (SAP-C03)

**Question:**

A company runs a document-processing platform on Amazon EKS. An API service accepts independent customer requests and places processing jobs in an internal buffer. A dispatcher combines jobs from unrelated requests into batches and sends each batch to worker services through a proprietary messaging protocol. Workers process jobs concurrently and can retry individual jobs.

The services use AWS Distro for OpenTelemetry instrumentation and export spans through collectors to AWS X-Ray. HTTP operations appear correctly, but worker operations appear as disconnected traces. The proprietary protocol has no automatic instrumentation. Each job already includes a business job ID, and each batch has a batch ID.

Operations engineers must navigate from an individual API request to its worker operations, distinguish retry attempts, and identify which operation contributed to the processing delay. Jobs from unrelated requests must not be represented as descendants of the same request. The company must retain its batching protocol and vendor-neutral instrumentation.

Which solution meets these requirements?

**Correct answer:** Create a producer span for each job under its originating request context and inject that span's context into the job's metadata. At the worker, extract the context for each job, create a child processing span, and activate that context only for the job's execution. Record each retry attempt in a separate child span.

**Explanation:** Per-job context propagation preserves the originating request's trace and causal relationships despite batching. Extracting and activating context separately for concurrent jobs prevents context leakage between requests. Distinct retry spans make attempt durations and failures visible without conflating separate executions.

### Question 3 (DVA-C02)

**Question:**

A document-processing application uses several microservices. A small percentage of requests take more than 10 seconds, although most finish within 1 second. Amazon CloudWatch dashboards show service-level latency and error rates. Centralized logs record each service's completion status, but do not identify downstream calls or retries. A developer needs to investigate an individual slow request and determine which downstream operations ran, how they were related, and how much time each consumed. Which approach best meets this requirement?

**Correct answer:** Instrument service handlers and downstream calls to send related spans to AWS X-Ray, then inspect the slow request's trace timeline.

**Explanation:** Distributed tracing connects operations belonging to an individual request. Related spans expose the downstream call structure, timing, and instrumented retry attempts. This complements metrics that show performance trends and logs that record individual events.

### Question 4 (AIP-C01)

**Question:**

An enterprise operates a centralized generative AI gateway that performs authorization, retrieves approved context through a separate service, and invokes a model through an adapter service. A recent adapter release introduced intermittent latency. Engineers need complete request-level execution paths across these services to distinguish retrieval delays, authorization failures, and model-call delays.

The telemetry policy requires retaining complete traces for every request that exceeds 8 seconds or fails authorization, while retaining approximately 1% of other requests. Neither outcome is known when a request begins. Requests finish within 30 seconds, and all associated telemetry arrives within 45 seconds of request initiation. The telemetry infrastructure has sufficient processing and buffering capacity, but long-term storage is limited. Prompts, retrieved text, responses, and user identifiers must not enter the telemetry pipeline.

Which implementation meets these requirements?

**Correct answer:** Propagate trace context across all service calls and record all spans initially. Export only allowlisted operational attributes. Route spans with the same trace ID to the same collector decision point, buffer beyond the maximum telemetry arrival interval, and retain traces based on total duration, authorization outcome, or a 1% baseline sampling rule.

**Explanation:** This design preserves the complete execution path until the final outcome is available. Trace-aware routing and adequate buffering allow a collector to evaluate the complete trace before making a tail-sampling decision. Duration and authorization rules retain the required requests, while baseline sampling limits storage for other requests. Allowlisting attributes before export keeps prohibited content out of the telemetry pipeline.

### Question 5 (SAA-C03)

**Question:**

An insurance company runs a quote application across multiple Availability Zones. Each quote request passes through three HTTP microservices and a custom internal forwarding service that constructs a new HTTP request for each downstream call. The microservices already send request timing and error data to AWS X-Ray. During a resilience test, engineers find separate traces for each microservice but cannot follow a single customer request through the entire application. They need to connect the operations for each request, even when successive requests reach different replicas.

Which change will meet this requirement?

**Correct answer:** Ensure that the forwarding service and microservices extract incoming trace context and inject the corresponding context into downstream HTTP requests, preserving trace and parent-child relationships.

**Explanation:** Propagating trace context across every HTTP boundary allows X-Ray to connect the services' recorded operations into an end-to-end trace. The relationship follows the request rather than depending on which replica handles it.

### Question 6 (DOP-C02)

**Question:**

A company runs an order service and an inventory service on Amazon ECS. Both services use AWS X-Ray SDK middleware to record incoming HTTP requests. Each task has a working X-Ray daemon sidecar with the required permissions and network connectivity.

During a controlled test in which every request is sampled, an order request produces two separate traces: one for the order service and one for the inventory service. Both traces contain the same orderId annotation. The order service calls the inventory service through a custom HTTP client that has no tracing instrumentation. Its trace contains no subsegment for that call.

A DevOps engineer must make each tested order request appear as one connected trace that shows the service relationship and the time spent on the outbound inventory call. Which change meets these requirements?

**Correct answer:** Instrument the custom HTTP client to create an outbound subsegment and inject the current X-Ray trace context into the X-Amzn-Trace-Id header. Ensure that the inventory service middleware uses the incoming context when creating its segment.

**Explanation:** The outbound subsegment records the inventory call's duration. Propagating its trace context allows the inventory service segment to use the same trace ID and the appropriate parent relationship, connecting both services within one request trace.

## Practice Distributed Tracing in a full exam

Questions on Distributed Tracing in each practice exam, with every answer option and an explanation for each:

- [AIP-C01](https://blowstack.com/practice-exams/aws-certified-generative-ai-developer-professional): 12 questions
- [MLA-C02 (beta)](https://blowstack.com/practice-exams/aws-certified-machine-learning-engineer-associate): 8 questions
- [DVA-C02](https://blowstack.com/practice-exams/aws-certified-developer-associate): 8 questions
- [DOP-C02](https://blowstack.com/practice-exams/aws-certified-devops-engineer-professional): 7 questions
- [SAP-C03](https://blowstack.com/practice-exams/aws-certified-solutions-architect-professional): 4 questions
- [SAA-C03](https://blowstack.com/practice-exams/aws-certified-solutions-architect-associate): 2 questions

## What is Distributed Tracing

Distributed tracing records the path of a request through a distributed application. A checkout request, for example, might pass through an API gateway, an order service, an inventory service, and a payment provider. Each component can appear healthy in isolation while the complete checkout remains slow or fails. Tracing connects these operations so engineers can investigate the behavior experienced by the caller.

A trace contains spans, which represent individual operations such as handling an HTTP request, executing a database query, or calling another service. A span typically records its name, start time, duration, status, and attributes. A trace identifier associates related spans, while parent-child relationships describe how operations connect. Asynchronous workflows can also use span links to express relationships that do not fit a simple parent-child hierarchy.

Applications must propagate trace context across component boundaries, commonly through HTTP headers or message metadata. Instrumentation extracts incoming context, creates spans, and injects context into outgoing calls. Libraries and frameworks can automate much of this work, while custom instrumentation captures business operations that automatic instrumentation misses. OpenTelemetry provides vendor-neutral instrumentation and telemetry collection components. A tracing backend stores and analyzes the resulting data.

In a cloud application, engineers might instrument containerized services and serverless functions, then send their spans through a collector to a managed tracing backend. A trace could reveal that a two-second checkout spends most of its time waiting for an inventory lookup. Another might show repeated payment retries after an upstream timeout. These findings help engineers narrow an investigation, but a slow span does not by itself prove the underlying root cause.

Tracing complements rather than replaces metrics and logs. Metrics show trends such as rising error rates across thousands of requests. Traces show the execution of particular requests, and correlated logs provide detailed events from those executions. For AWS exam scenarios, AWS X-Ray is associated with tracing requests and analyzing application dependencies. Amazon CloudWatch provides complementary monitoring, logs, and application observability. AWS CloudTrail records account activity and API events; it is not a substitute for application request tracing.

Tracing introduces processing, network, storage, and operational costs. Sampling limits the requests retained, but it can omit rare failures, depending on the sampling policy. Broken context propagation creates incomplete traces, and unsupported libraries or external providers can leave visibility gaps. Trace attributes should exclude credentials, payment details, and unnecessary personal information. Finally, overlapping spans must not simply be added together: concurrent operations overlap in time, and asynchronous work may continue after the original response has returned.

## Learn Distributed Tracing in lessons

9 short lessons on Distributed Tracing in 6 certification courses.

- CLF-C02: Distributed Tracing: following requests across services (Premium), https://app.blowstack.com/lessons/distributed-tracing-following-requests-across-services-clf-c02
- MLA-C02 (beta): Distributed Tracing: interpreting workflow traces (Premium), https://app.blowstack.com/lessons/distributed-tracing-interpreting-workflow-traces-mla-c02
- SAP-C03: Distributed Tracing: trace continuity across service meshes (Premium), https://app.blowstack.com/lessons/distributed-tracing-trace-continuity-across-service-meshes-sap-c03
- AIP-C01: Event Correlation vs. Distributed Tracing (Premium), https://app.blowstack.com/lessons/event-correlation-vs-distributed-tracing-aip-c01
- SAA-C03: Distributed Tracing: end-to-end request visibility (Premium), https://app.blowstack.com/lessons/distributed-tracing-end-to-end-request-visibility-saa-c03
- DOP-C02: Distributed Tracing: analyzing request execution (Premium), https://app.blowstack.com/lessons/distributed-tracing-analyzing-request-execution-dop-c02
