# Canary Testing

Canary testing validates a candidate configuration by exposing it to a small share of production traffic before deciding whether to expand or reverse the rollout.

Cloud concept. Updated 2026-10-29. Page: https://blowstack.com/tools/cloud-concepts-hub/canary-testing

> A canary runs alongside the current production configuration so engineers can compare behavior under real workloads while limiting exposure to regressions. For generative AI applications, it helps validate changes to foundation models, prompts, retrieval settings, and inference parameters against quality, safety, latency, cost, and business criteria.

Practice exams covering Canary Testing: [AIP-C01 AWS Certified Generative AI Developer - Professional](https://blowstack.com/practice-exams/aws-certified-generative-ai-developer-professional).

## 6 key aspects of Canary Testing

Effective canary testing combines limited exposure, traceable configurations, representative evaluation, and explicit promotion or rollback criteria.

1. **Limited production exposure**: Route a small, controlled share of eligible traffic to the candidate while retaining the baseline. Increase exposure only after the candidate meets predefined acceptance criteria.
2. **Versioned configurations and stable cohorts**: Track the model, prompt, retrieval settings, and inference parameters used for each request. Keep users or conversations on a consistent configuration when switching would distort behavior or break continuity.
3. **Quality-aware cost and latency evaluation**: Compare token usage, inference cost, and response latency alongside task quality and safety. Prefer measures such as cost per successful task over token counts alone.
4. **Representative evidence**: Include important workload segments and gather enough observations to detect meaningful regressions. A small traffic percentage does not guarantee a representative sample or statistically reliable conclusions.
5. **Explicit promotion and rollback gates**: Define thresholds, observation windows, and decision owners before starting. Automate rollback for fast operational signals where practical, and incorporate slower quality or business evaluations before wider promotion.
6. **Layered evaluation**: Use offline multi-model evaluation to screen candidates, then validate selected configurations against real traffic. Combine infrastructure metrics with semantic evaluation and business outcomes because successful API calls do not establish answer quality.

## Cloud services for Canary Testing

Use Amazon Bedrock evaluation capabilities to compare foundation model candidates offline, then introduce selected configurations through application routing controlled by AWS AppConfig or an application deployment managed by AWS CodeDeploy. For models hosted on supported SageMaker AI real-time endpoints, deployment guardrails provide another mechanism for canary traffic shifting. Publish configuration-specific operational and evaluation metrics to Amazon CloudWatch, and connect suitable alarms to rollback controls. These services address different parts of the workflow: evaluation selects candidates, routing limits exposure, and monitoring supplies evidence. Application instrumentation and explicit quality criteria are still necessary to determine whether a candidate should receive more traffic.

- **[Amazon Bedrock](https://blowstack.com/tools/cloud-services-hub/amazon-bedrock)**: Amazon Bedrock provides foundation model inference and model evaluation capabilities that can help compare candidates before production exposure. For live canary testing, use application-level routing or an external rollout mechanism to select the model and configuration, then associate evaluation results with that selection.
- **[Amazon SageMaker](https://blowstack.com/tools/cloud-services-hub/amazon-sagemaker)**: For supported real-time endpoint deployments, SageMaker AI deployment guardrails support canary traffic shifting to a new fleet. A configured observation period and CloudWatch alarms help control promotion and rollback; semantic model-quality checks require additional evaluation instrumentation.
- **[AWS AppConfig](https://blowstack.com/tools/cloud-services-hub/aws-appconfig)**: AWS AppConfig can distribute versioned routing configuration and feature flags using controlled deployment strategies. An application can consume that configuration to choose a baseline or candidate model, while configured CloudWatch alarms can trigger configuration rollback.
- **[AWS CodeDeploy](https://blowstack.com/tools/cloud-services-hub/aws-codedeploy)**: AWS CodeDeploy supports canary traffic shifting for supported deployment targets, including AWS Lambda deployments. It can gradually expose a new application version containing changed model-routing logic or prompts, but it does not independently evaluate foundation model response quality.
- **[Amazon CloudWatch](https://blowstack.com/tools/cloud-services-hub/amazon-cloudwatch)**: Amazon CloudWatch collects operational metrics, logs, and application-published evaluation metrics for baseline and candidate configurations. Alarms can participate in deployment rollback, while dashboards help compare errors, latency, token usage, and quality signals by configuration.

## Canary Testing exam questions

Real practice exam questions on Canary Testing, with the correct answer and its explanation.

### Question 1 (AIP-C01)

**Question:**

A company operates a customer-support assistant on an auto-scaled application fleet. Offline evaluation has identified a promising candidate foundation model configuration. The application already supports dynamically loaded routing configuration, deterministic session-ID hashing, and pinning a selected configuration for the duration of a session.

The team wants to expose the candidate to approximately 5% of new sessions, independent of the number of application instances. During a 30-minute observation period, the system must automatically stop assigning new sessions to the candidate if its unsupported-answer rate exceeds a defined threshold. Existing sessions can finish with their assigned configuration. The team publishes candidate-specific quality metrics to Amazon CloudWatch and wants to implement this test without deploying new application code.

Which solution best meets these requirements?

**Correct answer:** Use AWS AppConfig to distribute a versioned configuration containing the baseline settings, candidate settings, and a 5% session-routing threshold to the entire fleet. Use the application's session-ID hashing and session pinning for assignment. Configure a 30-minute final bake time and associate a candidate-quality CloudWatch alarm with the environment so an alarm rolls back to the previous configuration, which assigns no new sessions to the candidate.

**Explanation:** Application-level session routing controls exposure independently of fleet size, while pinning preserves conversation continuity. AppConfig distributes the configuration without an application code deployment. Its CloudWatch alarm integration can automatically roll back during deployment monitoring, including the final bake time. Restoring the previous routing configuration stops new candidate assignments while existing sessions retain their pinned settings.

## Practice Canary Testing in a full exam

Questions on Canary Testing in each practice exam, with every answer option and an explanation for each:

- [AIP-C01](https://blowstack.com/practice-exams/aws-certified-generative-ai-developer-professional): 2 questions

## What is Canary Testing

Canary testing is a controlled production evaluation technique. Instead of replacing a working configuration for every user, engineers direct a small portion of traffic to a candidate while the existing configuration handles the remainder. They observe predefined metrics, compare the candidate with the baseline, and either increase exposure, continue gathering evidence, or roll back. The goal is to discover production-specific problems while limiting their impact, not to prove that a release is risk-free.

In a cloud application, a router, feature flag, load balancer, or deployment controller can assign traffic to the candidate. For a generative AI application, the candidate might use a different foundation model, prompt template, retrieval configuration, or maximum output length. Version the complete configuration and record the assigned version with each evaluation event. When conversation continuity matters, use stable assignment by user or session so a single conversation does not unexpectedly switch models or prompts.

Start with systematic offline evaluation before exposing users. Run several models and configurations against representative datasets using task-specific scoring, human review, or calibrated model-based judges. Include difficult examples, safety tests, and relevant business scenarios. Amazon Bedrock model evaluation can support this screening, but offline evaluation does not itself route live traffic. Select promising candidates using quality requirements and cost-performance trade-offs, then use a canary to check whether those results hold in production.

For example, a support assistant could send 2% of eligible sessions to a candidate model while retaining the current model for the other sessions. Measure groundedness, task completion, harmful-output rates, escalation rates, time to first token, end-to-end latency, errors, and cost per successfully resolved case. Record input and output token usage separately because prices and usage patterns differ. Fewer tokens do not automatically mean better efficiency: a terse response that causes repeated questions can increase total cost and reduce customer satisfaction.

Define acceptance criteria before launch. A candidate might need to maintain a minimum quality score, keep high-percentile latency within a service objective, and reduce cost per successful task without increasing safety violations. Compare latency and cost at an acceptable quality level rather than selecting the fastest or cheapest model in isolation. Operational alarms can stop exposure quickly, while semantic quality and delayed business outcomes may require asynchronous evaluation and a longer observation window.

Canary testing overlaps with A/B testing but serves a different primary purpose. A/B testing typically estimates comparative outcomes through an experimental design; canary testing primarily limits rollout risk. A canary is not automatically a statistically valid experiment. Small samples, uneven customer segments, changing traffic patterns, and repeated examination of results can produce misleading conclusions. Choose representative cohorts, account for conversation-level correlation, and require enough observations before promoting a candidate. Shadow testing is also different: it copies requests to a candidate without returning its responses to users.

The main trade-offs are additional routing logic, evaluation expense, temporary parallel capacity, and incomplete coverage at low traffic volumes. Maintain baseline capacity for rollback, protect sensitive prompts and responses in evaluation logs, and make tool calls or other side effects safe. Switching traffic back cannot undo an incorrect transaction or a harmful response already delivered. Finally, synthetic monitoring checks sometimes called canaries are not the same as a production canary rollout; they test scripted interactions rather than exposing a live user cohort to a candidate.

## Learn Canary Testing in lessons

2 short lessons on Canary Testing in 1 certification course.

- AIP-C01: Canary Testing: evidence-gated production rollout (Premium), https://app.blowstack.com/lessons/canary-testing-evidence-gated-production-rollout-aip-c01
- AIP-C01: A/B Testing vs. Canary Testing: comparative evidence versus release risk (Premium), https://app.blowstack.com/lessons/a-b-testing-vs-canary-testing-comparative-evidence-versus-release-risk-aip-c01
