Cloud concept · Updated

Regression Testing - Regression testing checks that changes to a system do not break previously accepted behavior or degrade established quality standards.

In short

Regression testing reruns a maintained set of tests after changes and compares results with approved expectations or a trusted baseline. For generative AI applications, it helps detect unintended changes in answer quality, safety, tool use, latency, and cost when prompts, models, retrieval data, or application code change.

Cloud Concepts Hub →

6 key aspects of Regression Testing

Effective regression testing combines representative versioned cases, appropriate evaluators, controlled comparisons, and enforceable quality gates to detect unintended degradation.

01
Versioned cases and baselines

Maintain representative inputs, approved expectations, and baseline results alongside prompt and application versions. Record model settings, retrieval context, and evaluator versions so comparisons remain interpretable.

02
Assertions suited to generative outputs

Use exact checks for structured requirements and rubric-based evaluation for open-ended answers. Do not reject a correct answer merely because its wording differs from a reference.

03
Coverage of defects and edge cases

Include historical failures, missing context, conflicting instructions, tool errors, and adversarial requests. Convert confirmed production defects into permanent regression cases.

04
Controlled comparisons and variability

Run baseline and candidate versions against the same cases under controlled conditions. Repeat important cases and inspect category-level results to distinguish meaningful degradation from output variability.

05
Automated quality gates

Block deployment when critical assertions fail or quality, latency, and cost violate approved limits. Require reviewed evidence before changing thresholds or accepting a new baseline.

06
Continuous evaluation and diagnosis

Combine change-triggered tests with scheduled evaluations and appropriately protected production samples. Preserve per-case outputs and traces to isolate prompt, model, retrieval, or integration failures.

Cloud services for Regression Testing

A delivery workflow can use CodeBuild to start a regression suite with versioned test artifacts from S3. Step Functions coordinates application calls to models through Bedrock, while Lambda performs deterministic assertions and custom scoring. Results and configuration identifiers are saved to S3, and CloudWatch receives logs and quality metrics for dashboards and alarms. The build waits for evaluation completion and fails if approved acceptance rules are violated. Scheduled executions can reuse the same workflow to detect degradation between releases, with concurrency limits and protected datasets controlling cost and exposure.

Amazon Bedrock

Applications can invoke foundation models through Amazon Bedrock and compare responses across prompt or model configurations. Bedrock model evaluation capabilities can support quality assessment, while application-specific assertions and deployment acceptance rules remain part of the surrounding testing workflow.

AWS LambdaAWS Lambda

Lambda functions can invoke an application under test and validate schemas, required properties, tool selections, or custom quality rules. They can return structured per-case results and publish evaluation metrics without requiring a continuously running test server.

AWS Step FunctionsAWS Step Functions

Step Functions can orchestrate baseline and candidate runs, execute cases concurrently, coordinate evaluators, and route failures for review. Configure concurrency and retries carefully so model quotas are respected and transient infrastructure failures are distinguished from quality failures.

Amazon CloudWatchAmazon CloudWatch

CloudWatch stores logs and custom metrics such as schema failure rate, groundedness scores, and evaluation latency. Dashboards and alarms expose regression trends, but the evaluation workflow must calculate semantic quality scores.

Amazon S3Amazon S3

S3 can store test datasets, reference answers, baseline outputs, and evaluation reports. Object versioning and recorded version IDs help preserve the exact artifacts used for a comparison; access controls and retention policies protect sensitive test content.

AWS CodeBuildAWS CodeBuild

CodeBuild can run prompt testing frameworks and automated regression suites as part of a delivery workflow. A build can fail when evaluation results violate approved quality gates, preventing downstream deployment stages from proceeding when configured accordingly.

Regression Testing exam questions

Real practice exam questions on Regression Testing, with the correct answer and its explanation. Answer them with every option in the exam simulator.

DVA-C02

A developer maintains a regression test suite containing input events and expected JSON responses for an AWS Lambda function. After deploying a candidate version to a development environment, an AWS CodeBuild project must execute these tests against that version. The build must fail if any invocation encounters a function execution error or returns a response that differs from the expected response. Which approach meets these requirements?

Show answerCORRECT ANSWER

Invoke the candidate version with InvocationType set to RequestResponse. Check for invocation API failures and the FunctionError field, compare the returned payload with the expected JSON, and exit the test process with a nonzero status if any check fails.

RequestResponse invokes the deployed function synchronously and returns its execution result. A function execution error can occur even when the invocation API returns HTTP 200, so the tests must check FunctionError as well as API failures. Comparing the payload with the expected response validates application behavior, and a nonzero test-process exit status causes the CodeBuild command to fail.

SAP-C03

A logistics company maintains 35 applications built on .NET Framework. The company wants to move the application tiers to Linux containers on AWS with the least redevelopment effort. Several applications invoke the same vendor-supplied native Windows DLL to calculate shipping charges. The vendor will not provide a Linux version or source code. Security permits this DLL to remain on Windows behind an authenticated internal API, but the application tiers must run on Linux.

The applications have incomplete unit-test coverage. The company has representative production requests and approved shipping-charge results. Different development teams must modernize applications incrementally without changing externally visible behavior.

Which modernization strategy best meets these requirements?

Show answerCORRECT ANSWER

Use AWS Transform for .NET to assess compatibility and transform supported application code to cross-platform .NET. Expose the native DLL through a Windows-hosted internal service and update dependent applications to call that service. Review generated changes and tests, and validate each application's Linux deployment against the approved request-and-result baseline before release.

AWS Transform can assess and modernize supported .NET Framework code, but unsupported native Windows dependencies require separate remediation. An authenticated Windows-hosted service preserves the vendor implementation while allowing the application tiers to run on Linux. Human review and regression testing against approved results verify behavior during incremental releases; generated unit tests alone do not establish equivalence.

AIP-C01

A retailer uses Amazon Bedrock for a returns-policy assistant. Its versioned predeployment test suite includes routine requests, boundary cases, and previously corrected defects. Tests check policy decisions and required explanations rather than exact response wording.

The retailer is changing the return window for unopened electronics from 30 days to 14 days. All other return rules remain unchanged. A candidate prompt and retrieval snapshot implement the new policy. The current production configuration correctly implements the old policy, so some previously approved answers must now change.

The developer must validate the intended policy change while detecting damage to previously working behavior. The maintained suite must continue to cover electronics boundary cases and historical defects. Which approach BEST meets these requirements?

Show answerCORRECT ANSWER

Version the suite for the new policy. Have policy specialists revise expectations for affected cases and add new boundary cases while retaining unaffected cases and historical defects. Evaluate affected cases against the new policy, and compare candidate and production behavior on unaffected cases using controlled inputs and predefined tolerances.

This separates intentional behavioral changes from unintended regressions. Independently approved expectations validate the new requirements, while retained cases and baseline comparisons protect behavior that should not change. Versioning preserves the meaning of earlier test results.

Practice Regression Testing in a full exam

Questions on Regression Testing in each practice exam, with every answer option and an explanation for each:

Start a free practice exam

What is Regression Testing

3 min read

Regression testing answers a specific question: did a change damage behavior that previously worked? It differs from testing whether a new feature works, although both can run in the same workflow. In a generative AI application, the tested system includes more than the foundation model. System instructions, prompt templates, inference parameters, retrieval configuration, source documents, tools, guardrails, and response-processing code can all affect the result.

Start with a versioned test dataset that represents accepted application behavior. Each case should contain the input, relevant context, expected properties, and evaluation criteria. Include common requests, previously reported defects, ambiguous instructions, long conversations, missing retrieval results, malformed tool responses, and adversarial inputs. For a support assistant, a case might require an answer grounded in a policy document, a valid citation, and escalation when the policy does not contain an answer. Synthetic data can improve coverage, but sanitized production examples help reveal realistic failure patterns.

Generative outputs usually should not be judged by exact string equality. A correct response can use different wording, and low temperature does not guarantee identical results. Use deterministic assertions for requirements such as JSON schema validity, required fields, allowed tool names, and numeric calculations. Evaluate open-ended responses with task-specific rubrics for correctness, relevance, groundedness, and safety. Model-based judges can increase evaluation coverage, but their scores require calibration against human judgments; they can be biased or affected by adversarial content in the responses they evaluate.

Compare candidate and baseline versions using the same cases and controlled configurations. Record prompt versions, model identifiers, inference settings, retrieval inputs, evaluator versions, and application revisions so failures can be investigated. Where retrieval changes are intentional, test both fixed-context prompt behavior and end-to-end retrieval behavior to distinguish their effects. Repeat selected cases to estimate variability rather than interpreting one unusual answer as proof of a regression. Examine results by category because an improved overall average can hide a serious decline in a critical workflow.

Cloud implementations commonly use automated workflows to invoke the application, validate responses, store evidence, and enforce deployment gates. For example, AWS Step Functions can coordinate edge-case tests and parallel evaluations, while AWS Lambda functions check response schemas and expected properties. Amazon CloudWatch can collect the resulting custom quality metrics and trigger alarms. CloudWatch does not independently understand prompt quality or perform semantic regression evaluation; application code or an evaluation system must produce those measurements.

Define acceptance rules before running the candidate. A deployment might require every critical authorization test to pass, groundedness to remain above an approved threshold, and latency and token cost to stay within budgets. Compare against both absolute standards and baseline performance, since an already-poor baseline is not a sufficient quality target. A failed gate should preserve diagnostic artifacts and block promotion pending investigation. Scheduled evaluations and monitored production samples complement deployment tests by detecting changes in external dependencies or input patterns.

When a regression appears, compare individual failing cases and change one relevant factor at a time. A prompt version comparison may reveal conflicting instructions, weakened examples, or lost constraints, while retrieval traces may show that the prompt is not the cause. Add confirmed defects to the suite and require review before accepting new baselines. Larger suites improve coverage but increase inference cost and runtime, so run fast critical checks on every change and broader repeated evaluations before release. Passing a regression suite reduces known risks; it does not prove correctness for every possible request or replace security testing and human review.

Learn Regression Testing in lessons

3 short lessons on Regression Testing in 2 certification courses.

Open the lessons