Quality Gates

A quality gate evaluates evidence against predefined acceptance criteria before allowing a workflow to proceed. The evidence might include test results, vulnerability findings, performance measurements, foundation model evaluations, or production health metrics. Running tests alone does not create a gate: the workflow must interpret the results and block promotion, request review, or initiate a defined recovery action when requirements are not met.

Place gates where they provide useful feedback at an appropriate cost. Before merging code or during an initial build, run fast unit tests, static analysis, dependency checks, and infrastructure template validation. After deploying to an isolated test environment, run integration and contract tests against real service interfaces, including authentication and authorization behavior. Reserve broader end-to-end, load, resilience, and dynamic security tests for suitable staging environments or scheduled workflows because these tests generally require more time and resources. Critical release requirements must still be satisfied before production promotion; scheduling a test separately does not make its result optional.

For example, an API pipeline might reject a build when unit tests fail or a prohibited critical vulnerability is found. After staging deployment, another gate could require checkout integration tests to pass and measured request latency to remain within an agreed budget under representative load. A production canary can then expose a small traffic share while operational metrics are observed. Failed health criteria should stop further rollout and, where supported and configured, trigger rollback. A deployment completing successfully is not evidence that application behavior is correct.

For generative AI applications, treat changes to the foundation model, prompt templates, retrieval configuration, knowledge corpus, guardrails, and inference parameters as potential behavioral changes. Evaluate candidates against versioned datasets containing representative tasks, difficult cases, known failures, and adversarial inputs. Gate on task-relevant measures such as answer correctness, groundedness, retrieval relevance, unsafe response rates, sensitive-data disclosure, latency, and cost. For example, a support assistant might require minimum grounded-answer performance, no observed disclosure in a designated privacy test suite, and no material regression on critical customer workflows. Passing a finite test suite does not prove that disclosure can never occur.

Model outputs are often nondeterministic, so exact string matching is usually unsuitable for open-ended answers. Combine deterministic checks for schemas and tool arguments with semantic evaluation, rubric-based assessment, and targeted human review. Repeated trials and uncertainty estimates help distinguish meaningful regressions from sampling variation. Model-based judges should be calibrated against human judgments and checked for bias and inconsistency. Do not let a strong average score hide failures in safety-critical categories or important user segments.

Continuous evaluation extends gates beyond the initial release. Run regression suites on proposed changes and periodically evaluate deployed behavior using appropriately protected production samples. Keep an untouched holdout set for unbiased comparisons, and add newly discovered failures to a maintained regression set. Production monitoring can identify distribution shifts or emerging problems and trigger investigation, reevaluation, or rollback; it does not replace predeployment validation.

Define thresholds, evidence freshness, missing-result handling, and exception procedures explicitly. A report that exists but is stale, incomplete, or associated with another artifact should not satisfy the gate. Record the exact artifact, environment, dataset, evaluator, and policy versions so decisions can be reproduced. Promote the same immutable application artifact between environments rather than rebuilding untested binaries.

Strict gates reduce release risk but can increase delivery time, compute cost, and false rejections. Use risk-based test selection, parallel execution, and controlled isolation of flaky tests without silently bypassing essential coverage. Human approval can complement automated evidence for high-impact changes, but approval alone is not proof of quality. Exceptions should be authorized, time-limited, and auditable rather than becoming permanent bypasses.