Trace Sampling

Trace sampling controls the volume of distributed tracing data collected from an application. A trace represents a request or operation across service boundaries, and spans represent individual units of work within that trace. Recording every request can generate substantial processing, network, and storage costs, particularly in high-throughput applications. Sampling preserves selected traces rather than retaining detailed execution data for every request.

Head sampling makes its decision near the beginning of a request, before the outcome is known. A probability-based sampler might select approximately 5% of incoming requests, while a rule-based sampler might use the request path, service name, or other attributes available at that point. For example, an application could sample a larger proportion of checkout requests than health checks. A 5% probability is a long-term expectation, not a guarantee that exactly five out of every hundred requests will be captured.

Tail sampling delays the retention decision until spans have reached a collector and enough information is available to evaluate the trace. It can preferentially retain traces containing errors or high latency. This improves diagnostic coverage for unusual outcomes, but requires buffering, additional processing, and routing related spans to the same decision-making collector. Tail sampling cannot recover spans that an upstream head sampler already discarded, and incomplete or late-arriving spans can affect its decisions.

Sampling should generally be coordinated across the request path. Services propagate trace identifiers, parent relationships, and sampling information through supported trace-context headers. Downstream instrumentation commonly follows the upstream sampling decision so that selected requests produce connected traces. Independently sampling every service can produce fragmented traces that show only part of a transaction. For asynchronous processing, verify that producers, messaging integrations, and consumers preserve the tracing context required by the instrumentation.

On AWS, AWS X-Ray supports centralized sampling rules that compatible instrumentation uses to decide which requests to trace. Rules can match request properties and combine a reservoir target with a fixed sampling rate for additional matching traffic. The reservoir helps retain visibility into low-volume operations, while the rate controls sampling as traffic grows. Rule priority matters when multiple rules match. Centralized reservoir allocation also differs from local fallback sampling, where independently operating application instances can each apply their own reservoir allowance.

A practical implementation might enable X-Ray tracing on an Amazon API Gateway REST API, instrument application code using OpenTelemetry with an appropriate exporter, and propagate context to downstream services. If the REST API invokes AWS Lambda, active tracing allows Lambda to contribute tracing data, while instrumentation inside the function adds detail about application operations and downstream calls. Lambda follows an upstream sampling decision when one is provided; its own sampling behavior is not configured by assigning it arbitrary X-Ray sampling rules. Enabling tracing alone does not guarantee that every request is captured or every internal operation is instrumented.

Sampling is not a substitute for metrics, structured logs, or audit records. A rare failure may never appear in a low-rate sample, and error-focused tail sampling creates a dataset that is intentionally biased toward failures. Use request metrics for reliable traffic counts and error rates, and correlate logs with trace identifiers where supported. During troubleshooting, temporarily increasing sampling for a narrowly scoped route can help, but monitor ingestion cost and collector capacity. If traces remain missing, also check instrumentation, context propagation, export configuration, network access, and IAM permissions rather than assuming sampling is responsible.