Trajectory Evaluation

Trajectory evaluation examines how an application built with a foundation model reaches an outcome. In an agent, the trajectory can include tool selections, tool inputs, returned observations, retries, and termination decisions. In retrieval-augmented generation, or RAG, it can include query rewriting, retrieval, filtering, and use of retrieved evidence. In a workflow, it can include routing decisions, intermediate model outputs, validation, and transitions between processing stages. The evaluation target is the application's observable execution, not access to a model's private internal reasoning.

This approach complements final-answer evaluation. A customer-support agent might produce an accurate refund confirmation while skipping a required eligibility check. An answer-only evaluator could accept the response, whereas trajectory evaluation would identify the missing prerequisite. Conversely, a correctly executed workflow might fail because an external API is unavailable. Inspecting the trajectory helps distinguish model decision errors from tool failures and infrastructure problems.

Engineers begin by defining representative tasks and acceptable behavior. A refund task might require retrieving the order, checking policy and authorization, obtaining confirmation, and only then submitting the refund. Evaluation can compare the recorded execution with a reference sequence, check that required actions occurred, or enforce constraints such as authorization preceding a state-changing operation. Exact sequence matching is useful for tightly prescribed processes, but it can unfairly reject valid alternative paths. Dependency-aware or rubric-based evaluation is often more appropriate when several execution paths can succeed.

In cloud applications, instrumentation records correlated events across model requests, retrieval operations, functions, and workflow states. Useful fields include a task identifier, tool name, sanitized arguments, result status, timestamps, and model or prompt version. A test harness can apply deterministic checks to these records, while human reviewers or model-based judges assess contextual questions such as whether a selected tool was appropriate. Model-based judgments require clear rubrics and calibration against human-reviewed examples; they are not automatically reliable ground truth.

For a RAG assistant answering a policy question, trajectory evaluation can reveal that retrieval returned relevant documents but a later step discarded the authoritative version. For an infrastructure agent, it can detect repeated unsuccessful API calls, an unnecessary privileged operation, or failure to stop after completing the task. Practical measures include required-step completion, valid tool arguments, ordering violations, redundant calls, recovery behavior, latency, and cost. These should be considered alongside answer correctness, groundedness, and task success.

The main trade-offs are instrumentation effort, evaluation cost, and sensitivity to implementation changes. Detailed traces may contain confidential data, so access controls, redaction, and retention limits matter. Repeat tests because model-driven paths can vary, and version the application configuration and test dependencies for meaningful comparisons. Do not assume that fewer steps always means better performance: verification and approval steps may be essential. Finally, evaluation detects failures but does not prevent them; runtime authorization, input validation, and approval controls must independently enforce safety requirements.