Canary Testing

Canary testing is a controlled production evaluation technique. Instead of replacing a working configuration for every user, engineers direct a small portion of traffic to a candidate while the existing configuration handles the remainder. They observe predefined metrics, compare the candidate with the baseline, and either increase exposure, continue gathering evidence, or roll back. The goal is to discover production-specific problems while limiting their impact, not to prove that a release is risk-free.

In a cloud application, a router, feature flag, load balancer, or deployment controller can assign traffic to the candidate. For a generative AI application, the candidate might use a different foundation model, prompt template, retrieval configuration, or maximum output length. Version the complete configuration and record the assigned version with each evaluation event. When conversation continuity matters, use stable assignment by user or session so a single conversation does not unexpectedly switch models or prompts.

Start with systematic offline evaluation before exposing users. Run several models and configurations against representative datasets using task-specific scoring, human review, or calibrated model-based judges. Include difficult examples, safety tests, and relevant business scenarios. Amazon Bedrock model evaluation can support this screening, but offline evaluation does not itself route live traffic. Select promising candidates using quality requirements and cost-performance trade-offs, then use a canary to check whether those results hold in production.

For example, a support assistant could send 2% of eligible sessions to a candidate model while retaining the current model for the other sessions. Measure groundedness, task completion, harmful-output rates, escalation rates, time to first token, end-to-end latency, errors, and cost per successfully resolved case. Record input and output token usage separately because prices and usage patterns differ. Fewer tokens do not automatically mean better efficiency: a terse response that causes repeated questions can increase total cost and reduce customer satisfaction.

Define acceptance criteria before launch. A candidate might need to maintain a minimum quality score, keep high-percentile latency within a service objective, and reduce cost per successful task without increasing safety violations. Compare latency and cost at an acceptable quality level rather than selecting the fastest or cheapest model in isolation. Operational alarms can stop exposure quickly, while semantic quality and delayed business outcomes may require asynchronous evaluation and a longer observation window.

Canary testing overlaps with A/B testing but serves a different primary purpose. A/B testing typically estimates comparative outcomes through an experimental design; canary testing primarily limits rollout risk. A canary is not automatically a statistically valid experiment. Small samples, uneven customer segments, changing traffic patterns, and repeated examination of results can produce misleading conclusions. Choose representative cohorts, account for conversation-level correlation, and require enough observations before promoting a candidate. Shadow testing is also different: it copies requests to a candidate without returning its responses to users.

The main trade-offs are additional routing logic, evaluation expense, temporary parallel capacity, and incomplete coverage at low traffic volumes. Maintain baseline capacity for rollback, protect sensitive prompts and responses in evaluation logs, and make tool calls or other side effects safe. Switching traffic back cannot undo an incorrect transaction or a harmful response already delivered. Finally, synthetic monitoring checks sometimes called canaries are not the same as a production canary rollout; they test scripted interactions rather than exposing a live user cohort to a candidate.