End-to-End Testing

End-to-end testing evaluates an application as an integrated system. A test starts at a realistic entry point, such as a chat interface or API, and verifies the resulting response, application state, and external effects. In a cloud application, this path may cross authentication, networking, serverless functions, databases, model endpoints, and third-party APIs. The purpose is to establish whether the complete journey works, not merely whether each component passes its individual tests.

For applications built with foundation models, model performance and application performance are different questions. A model can score well on a benchmark while the application produces incorrect answers because it retrieves outdated documents, constructs an incomplete prompt, or mishandles tool results. End-to-end evaluation therefore uses representative tasks and measures the combined system against explicit acceptance criteria. These criteria can include task completion, factual correctness, groundedness, safety, authorization, response time, and cost per completed task.

Consider a retrieval-augmented generation application answering questions about employee benefits. An end-to-end test submits a question through the application, exercises identity-based document access and retrieval, invokes the model, and checks the final answer and citations. A successful answer should correctly address the question using evidence the requesting user is permitted to access. Tests should also cover missing evidence, contradictory documents, stale content, and instructions embedded in retrieved material that attempt to redirect the model. Retrieval metrics help diagnose failures, but strong retrieval scores alone do not prove that the final answer is correct.

For an agent that changes a delivery address, testing must go beyond checking the conversational response. The test should verify that the agent identifies the correct order, obtains any required confirmation, calls an authorized tool with valid arguments, and actually updates the intended record. It should also confirm that retries do not create unintended duplicate actions and that a failed tool call does not produce a false claim of success. Use sandbox accounts and synthetic records for actions that could otherwise affect customers or spend money.

Multi-step workflows require similar outcome-based checks. A document-processing workflow might extract information, classify a request, route it for approval, and store a result. End-to-end tests should exercise normal completion as well as timeouts, malformed model output, throttling, and dependency failures. Verify that fallback paths preserve business rules and that incomplete operations leave an understandable, recoverable state.

Build a versioned evaluation dataset containing realistic inputs, expected outcomes or scoring rubrics, and difficult boundary cases. Use deterministic assertions for structured outputs, permissions, state changes, and required fields. Use human review or a calibrated model-based judge for qualities such as relevance and groundedness, recognizing that automated judges can be biased or incorrect. Because generated responses can vary, assess repeated runs and aggregate success rates rather than requiring identical wording. Record model, prompt, retrieval, and configuration versions so regressions can be investigated.

In practice, run a focused suite during deployment and broader evaluations on a schedule or before significant changes. Capture correlated logs, traces, latency, token usage, and tool activity while protecting sensitive data. End-to-end tests are slower, more expensive, and harder to diagnose than unit tests, so they complement rather than replace component and integration tests. Mocking every dependency weakens confidence in real integration, while testing directly against production can introduce risk. A production-like isolated environment offers a useful balance, with controlled production checks where appropriate. Passing a finite test suite does not guarantee that a probabilistic application will always behave correctly; ongoing monitoring and feedback remain necessary.