Cloud concept · Updated

A/B Testing - A/B testing is a controlled experiment that randomly assigns comparable users or requests to two variants and measures which performs better against predefined criteria.

In short

A/B testing compares a baseline with an alternative under similar conditions to determine whether a change improves outcomes. For foundation model applications, it helps engineers evaluate changes to models, prompts, or retrieval configurations using real-world task success, human feedback, safety, latency, and cost rather than relying only on benchmark scores.

Cloud Concepts Hub →

6 key aspects of A/B Testing

Reliable A/B testing combines controlled assignment, reproducible variants, predefined outcomes, statistical discipline, and operational safeguards.

01
Randomized, consistent assignment

Assign users, sessions, or independent requests randomly to reduce selection bias. Keep assignment stable when interactions share context, and analyze results at an appropriate level.

02
A defined baseline and treatment

Compare the current implementation with a clearly specified alternative. Version models, prompts, retrieval settings, and inference parameters so the tested configurations can be reproduced.

03
Predefined metrics and safety limits

Choose a primary outcome such as task completion and supporting measures such as latency, cost, and factual accuracy. Establish safety thresholds that can stop an experiment even when its primary metric improves.

04
Statistically sound decisions

Plan sample size, experiment duration, and stopping rules before collecting results. Evaluate uncertainty and meaningful effect size rather than promoting a variant solely because its observed average is higher.

05
Complementary model evaluation

Use benchmark datasets and structured human review to screen foundation model variants before production exposure. Live A/B testing then measures outcomes that offline evaluation cannot fully reproduce.

06
Controlled exposure and observability

Record variant identifiers alongside outcomes, errors, latency, and usage costs. Maintain rollback controls and avoid collecting unnecessary personal or sensitive information.

Cloud services for A/B Testing

Amazon Bedrock can support offline foundation model evaluation, while Amazon SageMaker AI can distribute inference traffic across production variants for models hosted there. They are alternative model-serving paths rather than mandatory parts of the same deployment. Application logic establishes experiment assignment and records variant identifiers with user outcomes. Amazon CloudWatch tracks operational metrics and safety thresholds, while Amazon S3 retains datasets and evaluation artifacts for analysis. Together, these services support the infrastructure and evidence collection for an experiment, but they do not replace sound statistical design or structured human judgment.

Amazon Bedrock

Amazon Bedrock provides foundation model access and model evaluation capabilities, including automated and human evaluation workflows. Use these capabilities to assess candidate quality before a live experiment; production cohort assignment and outcome measurement still require an experiment design and application instrumentation.

Amazon SageMakerAmazon SageMaker

Amazon SageMaker AI real-time endpoints support production variants with configurable traffic weights, enabling comparisons between hosted model variants. Weighted routing does not by itself provide stable user assignment or statistical analysis, so applications must implement those requirements when needed.

Amazon CloudWatchAmazon CloudWatch

Amazon CloudWatch can collect logs and custom metrics labeled by experiment and variant, including latency, failures, and task outcomes. Dashboards and alarms help monitor safety thresholds, but experiment significance and causal interpretation require a separate analysis.

Amazon S3Amazon S3

Amazon S3 can store versioned evaluation datasets, model outputs, human review results, and experiment records for reproducible analysis. Apply access controls, encryption, and retention policies to protect sensitive prompts and responses.

A/B Testing exam questions

Real practice exam questions on A/B Testing, with the correct answer and its explanation. Answer them with every option in the exam simulator.

AIB-C01 (beta)

A nonprofit plans to test AI-personalized donation appeals against its existing appeal. Leadership considers a 10% relative increase in donations per eligible recipient sufficient to justify adoption. Donations are infrequent, response rates vary by day of the week, and recipients can donate within 14 days after receiving an appeal. The campaign manager proposes testing for one week because the mailing list is large. Which measurement plan should the business strategist establish before enabling AI to obtain reliable evidence of incremental value?

Show answerCORRECT ANSWER

Measure the existing appeal's donation rate over a representative period. Use that baseline and the required improvement to plan the sample size and test duration, randomly assign recipients to concurrent variants, and analyze results after the planned sample has completed its response window.

The preimplementation baseline supports planning an adequately sized experiment for the business-relevant improvement. Concurrent random assignment reduces systematic differences between groups, and allowing the response window to finish avoids comparing incomplete outcomes. A predefined analysis point also reduces the risk of declaring success from random fluctuations.

MLA-C02 (beta)

A company runs an A/B test of two document-routing models deployed as production variants on an Amazon SageMaker AI real-time endpoint. Independent documents are randomly assigned equally to the variants, and reviewers provide final correctness labels.

Before launch, the company specifies that replacing the control requires statistical evidence, at a one-sided significance level of 0.05, that the candidate improves accuracy by more than 0.3 percentage points. The experiment has reached its prespecified sample size and final analysis.

The candidate's observed accuracy is 0.4 percentage points higher than the control's. The one-sided 95% lower confidence bound for the improvement is 0.1 percentage points. A separate test against zero improvement produces a p-value below 0.05. All operational guardrails pass.

Which deployment decision is justified by these results?

Show answerCORRECT ANSWER

Retain the control because the results do not establish that the candidate's improvement exceeds 0.3 percentage points at the specified significance level.

The lower confidence bound is only 0.1 percentage points, below the required 0.3-percentage-point improvement. The results support some improvement but do not satisfy the prespecified deployment criterion.

AIP-C01

An insurance company uses Amazon Bedrock to generate plain-language explanations of policy documents. A revised prompt has passed offline safety testing. The company starts a concurrent production experiment that randomly assigns each customer to the current or revised prompt, with assignment balanced within each supported language. All other generation settings remain unchanged.

The approved experiment plan requires at least 14 days of observation and a predetermined sample size for every language in each variant. Promotion requires confidence bounds to demonstrate that the revised prompt reduces the disparity in explanation-comprehension rates across languages without decreasing comprehension for any language beyond an approved tolerance.

On day 5, overall comprehension is significantly higher with the revised prompt. The estimated disparity is smaller, but one low-volume language has reached only 15% of its required sample size. Its confidence interval includes both a substantial improvement and a decline beyond the approved tolerance. A product manager requests immediate promotion because no language shows a statistically significant decline.

What should the developer do?

Show answerCORRECT ANSWER

Keep the existing randomized allocation and configurations unchanged until the minimum duration and per-language sample requirements are met. Then apply the planned disparity and per-language confidence-bound criteria before deciding whether to promote.

This preserves the controlled comparison and follows the predefined decision procedure. Adequate evidence is required for every language, not just the pooled population. The revised prompt must demonstrate both reduced disparity and acceptable comprehension within each language before promotion.

Practice A/B Testing in a full exam

Questions on A/B Testing in each practice exam, with every answer option and an explanation for each:

Start a free practice exam

What is A/B Testing

3 min read

A/B testing evaluates two alternatives through a controlled experiment. Variant A is usually the current implementation, while variant B introduces a defined change. Participants or requests are randomly assigned to a variant, and outcomes are compared using metrics selected before the experiment begins. Random assignment reduces systematic differences between groups, making it easier to attribute observed changes to the tested intervention rather than to unrelated factors.

In cloud applications, an application router, feature flag system, or inference endpoint can distribute traffic between variants. Assignment should match the unit of analysis. For a conversational assistant, assigning each user or conversation consistently to one variant prevents the experience from changing halfway through an interaction. Request-level assignment can be appropriate for independent tasks, but treating repeated requests from the same user as independent observations can overstate statistical confidence.

For foundation models, engineers might compare two models, two prompt templates, or two retrieval configurations. For example, a support assistant could compare its existing prompt with a revised prompt that requires answers grounded in retrieved documentation. The primary metric might be successful issue resolution, with unsupported-answer rate, escalation rate, response latency, and cost per resolved issue as additional measures. Keeping other configuration settings stable makes the result easier to interpret. If several components change together, the experiment evaluates the combined change rather than identifying which component caused the improvement.

Offline evaluation should usually precede live experimentation. Representative benchmark datasets provide repeatable comparisons of task quality, including accuracy where reference answers exist. Human-in-the-loop evaluation can assess qualities that automated scores miss, such as usefulness, factual support, appropriateness, and instruction following. Reviewers should use a consistent rubric, and hiding model identities or randomizing answer order can reduce evaluation bias. Amazon Bedrock model evaluation capabilities support automated evaluation and human evaluation workflows, but an offline comparison is not automatically a randomized production A/B test.

A sound experiment defines its hypothesis, primary metric, minimum meaningful improvement, sample-size assumptions, and stopping rule in advance. Results should account for uncertainty and practical significance: a statistically detectable improvement may be too small to justify higher inference costs. Repeatedly checking results and stopping at the first favorable difference can increase false-positive conclusions unless the analysis uses an appropriate sequential-testing method. Experiments should also run long enough to capture relevant usage patterns, such as weekday and weekend differences.

A/B testing has operational and ethical trade-offs. Running two variants increases infrastructure, inference, and analysis costs, while exposing users to an inferior variant can cause harm. Apply safety checks before launch, limit initial exposure, protect sensitive data in evaluation logs, and define rollback thresholds. Unequal traffic allocation is valid when planned appropriately; a 50/50 split is not mandatory.

Common misconceptions include treating traffic splitting alone as an experiment or assuming the variant with the highest average score is necessarily better. A canary deployment primarily limits release risk, whereas an A/B test measures a defined comparative outcome. Likewise, shadow testing sends copied traffic to a candidate without exposing its responses to users, so it cannot directly measure how those responses affect user behavior. Benchmarks, human review, and live A/B tests provide complementary evidence rather than interchangeable answers.

Learn A/B Testing in lessons

7 short lessons on A/B Testing in 3 certification courses.

and 1 more lesson

Open the lessons