Hypothesis Testing

Hypothesis testing starts with a precise question, a measurable outcome, and competing statistical claims. For example, an engineering team might ask whether a new recommendation model increases purchase conversion compared with the production model. For a prespecified one-sided superiority test, the null hypothesis states that the new model provides no improvement, while the alternative states that it improves conversion. Choose the direction of the test before examining results; a two-sided test instead evaluates differences in either direction.

In a production A/B test, the existing model serves a control group and the candidate model serves a treatment group. Random assignment helps make the groups comparable so that differences in outcomes can be attributed to the model change. Choose the assignment unit carefully: when users make repeated requests, stable user-level assignment usually prevents them from switching treatments. Statistical analysis must also account for dependence among observations from the same user rather than treating every request as independent.

Cloud inference platforms can host multiple model variants and distribute requests using traffic weights. However, weighted routing alone does not guarantee stable user cohorts. Applications may need deterministic assignment and explicit variant targeting. Record the experiment identifier, assigned variant, model version, prediction timestamp, and outcome identifiers so that predictions can be joined with delayed labels or business events. Latency and error rates are immediately observable, but accuracy, fraud losses, and conversion often require later ground truth.

Before launching, define the primary metric, significance level, minimum detectable effect, target statistical power, and analysis plan. Sample-size requirements depend on baseline performance, outcome variability, allocation, and the effect size worth detecting. A small treatment allocation limits exposure to a harmful model but typically increases the time needed to establish an improvement. The experiment should also cover relevant business cycles and allow enough time for outcomes to mature.

A p-value measures how incompatible the observed result is with the null hypothesis under the test's assumptions: it is the probability, assuming the null, of a result at least as extreme as the one observed. It is not the probability that the null hypothesis is true. Compare the p-value with the prespecified significance level, but also examine the estimated effect and its confidence interval. An improvement from 10.0% to 10.1% conversion may be statistically significant with sufficient traffic yet still fail to justify additional inference cost.

Select a statistical method appropriate to the metric and experimental design. A two-proportion test can suit independent binary conversion outcomes, while continuous or heavily skewed metrics may require other methods. Repeatedly checking ordinary fixed-horizon p-values and stopping when significance appears inflates false-positive risk. Use a predefined analysis point or a valid sequential method, and address multiple comparisons when testing many variants or outcomes.

Monitor guardrails such as tail latency, request failures, cost, and harmful predictions throughout the experiment. Operational safety thresholds can justify stopping exposure before the statistical analysis is complete. Conversely, a nonsignificant result does not establish equivalence; it may indicate insufficient evidence or inadequate power. Shadow testing cannot directly measure user responses to candidate predictions because those predictions are not served to users, and drift monitoring alone does not establish which model performs better. Promote a candidate only when the evidence, practical benefit, and guardrails support the decision, then continue monitoring after rollout.