Shadow Testing

Shadow testing separates evaluation from serving decisions. The production system continues handling user requests, while a copy of some or all requests is sent to a candidate implementation, often called a shadow variant. The candidate executes the workload, but its responses are retained for evaluation rather than returned to users. This makes shadow testing useful for validating new machine learning models, inference containers, preprocessing code, or infrastructure configurations against realistic inputs.

In the cloud, request duplication can occur in an application gateway, a proxy, application code, or a managed inference service. Engineers provision the candidate with its own compute capacity, define a representative sampling policy, and collect outputs and operational measurements. The duplication path should avoid making the user request wait for the shadow response. Separate capacity, bounded queues, and timeouts help prevent a slow candidate from affecting production availability.

Useful comparisons require more than observing whether the candidate returns a successful response. Record correlation identifiers, model versions, relevant feature versions, and timestamps so that production and shadow results can be matched. Compare prediction disagreement, score distributions, error rates, throughput, resource utilization, and percentile latency. Use equivalent measurement boundaries when comparing latency, and account for differences in hardware, request volume, concurrency, and warm-up behavior. A lightly sampled candidate does not demonstrate that it can handle the full production workload.

For example, a payment-risk service can mirror a sample of transaction requests to a new classifier while the existing classifier continues making decisions. Engineers compare scores and later join predictions with confirmed fraud outcomes to evaluate precision, recall, and business cost. Disagreement with production is not proof that the candidate is worse: production predictions are a baseline, not ground truth. Labels can arrive late, and production decisions may influence which outcomes become observable.

For a foundation model application, shadow requests can help compare response latency, token consumption, formatting compliance, and safety behavior. Different generated wording is not necessarily a regression, so evaluation needs task-specific scoring, human review, or validated automated evaluators. External tool calls and other actions must be disabled or redirected to an isolated environment. Discarding a response does not prevent the candidate from sending an email, charging a payment, or modifying a database.

An automated delivery pipeline can deploy an approved candidate as a shadow, collect evidence over a defined observation window, and apply release gates. Gates might require sufficient samples, acceptable error rates and tail latency, and no material degradation on labeled quality metrics or important customer segments. A passing candidate can then proceed to a canary or blue/green rollout. A failing candidate should stop progressing and have its shadow traffic disabled or its deployment removed.

Shadow testing is not A/B testing or a canary deployment: those approaches expose some users to candidate responses and can measure resulting user behavior. Shadow testing cannot directly establish business impact when users never see the candidate output. It also adds inference cost and duplicates potentially sensitive data, requiring appropriate access controls, encryption, retention, and regional handling. After promotion, monitoring and rollback remain necessary because a successful shadow test does not guarantee production success. Preserve the previous model and deployment configuration, define rollback thresholds, and automate reversion where supported; do not assume that shadow testing itself provides rollback.