Cloud concept · Updated

End-to-End Testing - End-to-end testing validates a complete application journey, from its entry point through integrated components to the final response and observable effects.

In short

End-to-end testing checks whether an application achieves the intended user outcome when its components work together under realistic conditions. For foundation model applications, it evaluates the combined behavior of prompts, retrieval, models, tools, safeguards, and workflows rather than relying on model benchmarks alone.

Cloud Concepts Hub →

6 key aspects of End-to-End Testing

Effective end-to-end testing combines realistic journeys, measurable outcomes, safe execution, and repeatable evaluation of the complete application.

01
Complete user-journey coverage

Start at the application's real entry point and verify the final outcome across integrated dependencies. Include persisted state and external effects, not just the response displayed to the user.

02
Application-level evaluation criteria

Define success using business outcomes, answer quality, safety, latency, and cost. A foundation model benchmark does not establish that the surrounding RAG pipeline, agent, or workflow performs correctly.

03
Representative and adversarial scenarios

Maintain realistic tasks alongside cases involving missing evidence, unauthorized access, prompt injection, and dependency failures. For agents, include multi-turn interactions and actions that require confirmation.

04
Probabilistic output assessment

Use exact assertions where outputs are structured and scoring rubrics where multiple responses can be valid. Combine repeated runs, human review, and calibrated automated judging to assess variable model behavior.

05
Safe, production-like execution

Exercise realistic infrastructure and integrations in isolated environments with synthetic data and restricted permissions. Ensure test agents cannot accidentally modify customer records or perform uncontrolled transactions.

06
Regression detection and diagnosis

Version evaluation datasets and application configurations, then compare results after changes to models, prompts, retrieval, or tools. Correlated telemetry helps distinguish retrieval failures, reasoning errors, tool failures, and infrastructure problems.

Cloud services for End-to-End Testing

A test harness running in AWS CodeBuild can submit representative requests to an application that uses Amazon Bedrock and, where appropriate, AWS Step Functions. The harness checks generated answers, citations, workflow completion, and resulting state against versioned acceptance criteria. Amazon CloudWatch supplies logs and metrics for diagnosing failures and tracking operational thresholds; scheduled synthetic checks can add ongoing coverage. Bedrock evaluation capabilities support quality scoring, but they do not replace assertions about the complete application. Together, these services support repeatable testing while keeping model quality, integration correctness, and business outcomes distinct.

Amazon Bedrock

Amazon Bedrock provides foundation model inference and capabilities for building RAG and agent applications that can be exercised by end-to-end test harnesses. Its evaluation capabilities can support model and RAG quality assessment, but complete application tests must additionally verify authorization, tool side effects, workflow outcomes, and operational requirements.

AWS CodeBuildAWS CodeBuild

AWS CodeBuild can run custom test harnesses that submit requests to deployed application endpoints, evaluate responses, and assert downstream state. Build failures can prevent promotion when defined end-to-end acceptance thresholds are not met.

Amazon CloudWatchAmazon CloudWatch

Amazon CloudWatch collects application logs and metrics that help explain end-to-end test failures and measure latency or error rates. CloudWatch Synthetics can run scheduled canaries against supported application interfaces, while custom instrumentation records model and tool behavior.

AWS Step FunctionsAWS Step Functions

AWS Step Functions orchestrates multi-step workflows whose complete execution can be tested from input to terminal state. Execution histories help verify branches, retries, error handling, and the sequence of integrated service calls, with additional assertions needed for business outcomes.

End-to-End Testing exam questions

Real practice exam questions on End-to-End Testing, with the correct answer and its explanation. Answer them with every option in the exam simulator.

AIF-C01

A software company is evaluating an AI assistant that creates project workspaces by selecting and calling tools to create a repository, assign team permissions, and register the project. Identical requests sometimes produce different outcomes. The company wants to estimate how reliably a user request will succeed during one execution under the intended production settings, without a user restarting the task. Which evaluation plan BEST meets this objective?

Show answerCORRECT ANSWER

Run each representative request multiple times from an equivalent clean test state using production settings, and measure the percentage of executions that produce the required workspace and permissions.

Repeated independent executions capture variability in the agent's decisions and tool use. Resetting the test state prevents previous executions from influencing later results. Verifying actual workspace state measures successful task completion rather than a plausible completion message.

DEA-C01

A retailer uses AWS CodePipeline to release a data pipeline. Uploading a sales file to an Amazon S3 landing bucket generates an event that starts an AWS Step Functions workflow. The workflow runs an AWS Glue job and updates catalog metadata. Analysts query the resulting daily sales totals through an Amazon Athena view.

Previous releases passed transformation unit tests but failed because event routing was misconfigured or new output partitions were not available through the view. The team needs an automated preproduction check that validates the complete processing path and the totals available to analysts before allowing production deployment.

Which test should the team run in an AWS CodeBuild action against an isolated environment containing the candidate release?

Show answerCORRECT ANSWER

Upload a version-controlled sales fixture to the landing bucket through its normal ingestion interface. Poll the Athena view for that test's uniquely identified data until a bounded deadline, assert the expected daily totals, and fail the build if the assertions are not satisfied.

This exercises the actual ingestion event, workflow, transformation, catalog update, and analyst-facing query path. Checking known totals verifies the business outcome, while the deadline handles asynchronous processing without allowing a missing result to wait indefinitely. A failed CodeBuild action can block production promotion.

DVA-C02

A developer deploys a checkout API to a development environment. A recent defect caused the API to return HTTP 200 even when an order was not saved. The environment receives little user traffic. The developer must test the deployed endpoint every 5 minutes by authenticating, submitting a test order, and retrieving the order to verify its contents. Failed checks must generate an alert. Which solution meets these requirements with the least operational overhead?

Show answerCORRECT ANSWER

Create an Amazon CloudWatch Synthetics canary that runs every 5 minutes. Script the authentication, order submission, and retrieval requests, assert that the retrieved order matches the submitted order, and configure a CloudWatch alarm on canary failures.

A scripted canary can actively exercise the deployed API on a recurring schedule and validate response contents across multiple requests. Assertions detect the unsaved-order defect even when HTTP responses indicate success, and a CloudWatch alarm can initiate a notification when the canary fails.

Practice End-to-End Testing in a full exam

Questions on End-to-End Testing in each practice exam, with every answer option and an explanation for each:

Start a free practice exam

What is End-to-End Testing

3 min read

End-to-end testing evaluates an application as an integrated system. A test starts at a realistic entry point, such as a chat interface or API, and verifies the resulting response, application state, and external effects. In a cloud application, this path may cross authentication, networking, serverless functions, databases, model endpoints, and third-party APIs. The purpose is to establish whether the complete journey works, not merely whether each component passes its individual tests.

For applications built with foundation models, model performance and application performance are different questions. A model can score well on a benchmark while the application produces incorrect answers because it retrieves outdated documents, constructs an incomplete prompt, or mishandles tool results. End-to-end evaluation therefore uses representative tasks and measures the combined system against explicit acceptance criteria. These criteria can include task completion, factual correctness, groundedness, safety, authorization, response time, and cost per completed task.

Consider a retrieval-augmented generation application answering questions about employee benefits. An end-to-end test submits a question through the application, exercises identity-based document access and retrieval, invokes the model, and checks the final answer and citations. A successful answer should correctly address the question using evidence the requesting user is permitted to access. Tests should also cover missing evidence, contradictory documents, stale content, and instructions embedded in retrieved material that attempt to redirect the model. Retrieval metrics help diagnose failures, but strong retrieval scores alone do not prove that the final answer is correct.

For an agent that changes a delivery address, testing must go beyond checking the conversational response. The test should verify that the agent identifies the correct order, obtains any required confirmation, calls an authorized tool with valid arguments, and actually updates the intended record. It should also confirm that retries do not create unintended duplicate actions and that a failed tool call does not produce a false claim of success. Use sandbox accounts and synthetic records for actions that could otherwise affect customers or spend money.

Multi-step workflows require similar outcome-based checks. A document-processing workflow might extract information, classify a request, route it for approval, and store a result. End-to-end tests should exercise normal completion as well as timeouts, malformed model output, throttling, and dependency failures. Verify that fallback paths preserve business rules and that incomplete operations leave an understandable, recoverable state.

Build a versioned evaluation dataset containing realistic inputs, expected outcomes or scoring rubrics, and difficult boundary cases. Use deterministic assertions for structured outputs, permissions, state changes, and required fields. Use human review or a calibrated model-based judge for qualities such as relevance and groundedness, recognizing that automated judges can be biased or incorrect. Because generated responses can vary, assess repeated runs and aggregate success rates rather than requiring identical wording. Record model, prompt, retrieval, and configuration versions so regressions can be investigated.

In practice, run a focused suite during deployment and broader evaluations on a schedule or before significant changes. Capture correlated logs, traces, latency, token usage, and tool activity while protecting sensitive data. End-to-end tests are slower, more expensive, and harder to diagnose than unit tests, so they complement rather than replace component and integration tests. Mocking every dependency weakens confidence in real integration, while testing directly against production can introduce risk. A production-like isolated environment offers a useful balance, with controlled production checks where appropriate. Passing a finite test suite does not guarantee that a probabilistic application will always behave correctly; ongoing monitoring and feedback remain necessary.

Learn End-to-End Testing in lessons

7 short lessons on End-to-End Testing in 5 certification courses.

and 1 more lesson

Open the lessons