Cloud concept · Updated

Trajectory Evaluation - Trajectory evaluation assesses the sequence of observable steps a foundation-model application takes to complete a task, rather than judging only its final answer.

In short

A trajectory includes actions such as retrieving documents, selecting tools, supplying arguments, handling results, and deciding when to stop. Evaluating this sequence helps engineers detect unsafe actions, incorrect dependencies, unnecessary work, and intermediate failures that a plausible final response can conceal.

Cloud Concepts Hub →

6 key aspects of Trajectory Evaluation

Trajectory evaluation combines execution tracing, behavioral requirements, and calibrated scoring to assess whether an application follows a correct, safe, and efficient path to completion.

01
Observable execution as evidence

Capture tool calls, retrieval events, workflow transitions, and their results under a shared task identifier. Evaluate these observable events without assuming they reveal the model's private reasoning.

02
Reference paths and behavioral constraints

Compare executions with expected paths or check requirements such as mandatory actions and ordering dependencies. Allow equivalent successful paths when the task does not require one exact sequence.

03
Intermediate failure diagnosis

Identify whether a failure originated in retrieval, tool selection, argument construction, external dependencies, or workflow routing. This makes remediation more precise than a single final-answer score.

04
Safety and authorization checks

Assess whether approvals, permission checks, and validation occurred before sensitive actions. Use evaluation to find violations, while enforcing these requirements through independent runtime controls.

05
Quality and efficiency together

Measure redundant calls, retries, latency, and cost alongside task success and correctness. Avoid rewarding a shorter trajectory that omits necessary checks.

06
Repeatable, calibrated assessment

Use versioned test cases and application configurations, repeat nondeterministic tasks, and calibrate model-based judges against human review. Combine deterministic assertions with contextual assessment rather than relying on one scoring method.

Cloud services for Trajectory Evaluation

Amazon Bedrock supplies model and agent execution capabilities, while AWS Step Functions can coordinate explicit application stages and record workflow behavior. Amazon CloudWatch collects instrumented logs and evaluation metrics across these components. An evaluation harness correlates events for each task, checks required actions and ordering constraints, and optionally submits sanitized traces for human or model-based review. The resulting trajectory scores should be analyzed alongside final-answer quality, task success, latency, and cost. These services provide execution evidence and operational infrastructure; engineers must still define acceptable behavior, implement scoring logic, and protect sensitive trace data.

Amazon Bedrock

Agents for Amazon Bedrock can return trace information during agent invocation when tracing is enabled, exposing orchestration events and tool interactions for inspection. Engineers can use those events in a custom trajectory evaluation harness; collecting a trace does not itself establish that the execution was correct.

AWS Step FunctionsAWS Step Functions

AWS Step Functions orchestrates explicit workflows that can include model calls, validation, branching, and tool execution. Standard Workflow execution history provides evidence for evaluating state transitions, retries, and required dependencies; Express Workflows use configured CloudWatch logging for execution details.

Amazon CloudWatchAmazon CloudWatch

Amazon CloudWatch can collect application logs and custom metrics describing model calls, tool use, errors, and timing. With correlation identifiers and appropriate instrumentation, engineers can query execution events and monitor metrics produced by their trajectory evaluators.

Trajectory Evaluation exam questions

Real practice exam questions on Trajectory Evaluation, with the correct answer and its explanation. Answer them with every option in the exam simulator.

AIF-C01

A company uses an agent built with Amazon Bedrock to book business travel. The agent must verify a traveler's spending authorization before making a nonrefundable reservation. During testing, the agent produces accurate final itineraries, but reviewers suspect that it sometimes makes reservations before checking authorization. Which evaluation approach would BEST detect this problem?

Show answerCORRECT ANSWER

Evaluate recorded agent traces to verify that a successful authorization check for the traveler and purchase amount precedes each reservation action.

Trajectory evaluation examines observable actions, arguments, results, and dependencies. Reviewing Amazon Bedrock agent traces against this requirement can reveal reservations made before the necessary authorization, even when final itineraries are accurate.

MLA-C02 (beta)

A company uses a multi-agent application built with Amazon Bedrock to analyze insurance claims. A supervisor delegates document review to a specialist and payment calculations to another specialist, then combines their findings into a recommendation.

After an application update, some recommendations omit documented coverage exceptions. Model invocation errors and tool error rates remain unchanged. The application records each investigation's task assignments, specialist inputs and outputs, messages passed between agents, and final recommendation under a common investigation ID.

An ML engineer must determine whether the exceptions are first lost during document review, transfer between agents, or final synthesis. Which evaluation approach best meets this requirement?

Show answerCORRECT ANSWER

Evaluate recorded investigations with stage-specific rubrics for extracting exceptions, preserving them in handoff messages, and incorporating them into recommendations. Identify the earliest stage where the required information is missing or misrepresented.

Evaluating intermediate outputs and handoffs within the same execution reveals where information first becomes incorrect or incomplete. This distinguishes document-review failures from coordination and synthesis failures rather than judging only the final recommendation.

AIP-C01

A travel company uses Agents for Amazon Bedrock to reschedule reservations through action groups. Customers often omit required details initially and provide constraints only when asked. The agent can use several valid sequences of clarification questions and reservation lookups.

The current benchmark replays recorded customer messages at fixed turn numbers. After an agent update, these messages sometimes answer questions the agent did not ask. The benchmark still reports high success rates, but pilot users experience repeated clarification questions and reservations that violate their stated constraints.

The company needs a repeatable pre-release evaluation that measures autonomous task completion and unnecessary interaction without requiring a single prescribed conversation path. Test runs must not modify production reservations.

Which solution BEST meets these requirements?

Show answerCORRECT ANSWER

Use a calibrated, goal-conditioned customer simulator that responds to the agent's actual questions without volunteering hidden constraints. Run against resettable sandbox reservations, verify terminal reservation state against each scenario's goal, and score repeated questions and non-progressing actions from correlated execution traces using a validated rubric and fixed task budgets.

A closed-loop simulator exercises clarification and adaptation rather than supplying predetermined messages. Resettable sandbox state provides isolation and repeatability. Authoritative reservation-state checks establish actual task completion, while trace-based evaluation detects unnecessary interaction without requiring exact conversation matching. Fixed budgets prevent unbounded conversations from appearing successful.

Practice Trajectory Evaluation in a full exam

Questions on Trajectory Evaluation in each practice exam, with every answer option and an explanation for each:

Start a free practice exam

What is Trajectory Evaluation

2 min read

Trajectory evaluation examines how an application built with a foundation model reaches an outcome. In an agent, the trajectory can include tool selections, tool inputs, returned observations, retries, and termination decisions. In retrieval-augmented generation, or RAG, it can include query rewriting, retrieval, filtering, and use of retrieved evidence. In a workflow, it can include routing decisions, intermediate model outputs, validation, and transitions between processing stages. The evaluation target is the application's observable execution, not access to a model's private internal reasoning.

This approach complements final-answer evaluation. A customer-support agent might produce an accurate refund confirmation while skipping a required eligibility check. An answer-only evaluator could accept the response, whereas trajectory evaluation would identify the missing prerequisite. Conversely, a correctly executed workflow might fail because an external API is unavailable. Inspecting the trajectory helps distinguish model decision errors from tool failures and infrastructure problems.

Engineers begin by defining representative tasks and acceptable behavior. A refund task might require retrieving the order, checking policy and authorization, obtaining confirmation, and only then submitting the refund. Evaluation can compare the recorded execution with a reference sequence, check that required actions occurred, or enforce constraints such as authorization preceding a state-changing operation. Exact sequence matching is useful for tightly prescribed processes, but it can unfairly reject valid alternative paths. Dependency-aware or rubric-based evaluation is often more appropriate when several execution paths can succeed.

In cloud applications, instrumentation records correlated events across model requests, retrieval operations, functions, and workflow states. Useful fields include a task identifier, tool name, sanitized arguments, result status, timestamps, and model or prompt version. A test harness can apply deterministic checks to these records, while human reviewers or model-based judges assess contextual questions such as whether a selected tool was appropriate. Model-based judgments require clear rubrics and calibration against human-reviewed examples; they are not automatically reliable ground truth.

For a RAG assistant answering a policy question, trajectory evaluation can reveal that retrieval returned relevant documents but a later step discarded the authoritative version. For an infrastructure agent, it can detect repeated unsuccessful API calls, an unnecessary privileged operation, or failure to stop after completing the task. Practical measures include required-step completion, valid tool arguments, ordering violations, redundant calls, recovery behavior, latency, and cost. These should be considered alongside answer correctness, groundedness, and task success.

The main trade-offs are instrumentation effort, evaluation cost, and sensitivity to implementation changes. Detailed traces may contain confidential data, so access controls, redaction, and retention limits matter. Repeat tests because model-driven paths can vary, and version the application configuration and test dependencies for meaningful comparisons. Do not assume that fewer steps always means better performance: verification and approval steps may be essential. Finally, evaluation detects failures but does not prevent them; runtime authorization, input validation, and approval controls must independently enforce safety requirements.

Learn Trajectory Evaluation in lessons

3 short lessons on Trajectory Evaluation in 3 certification courses.

Open the lessons