Cloud concept · Updated

Model Evaluation - Model evaluation is the systematic assessment of an AI model’s quality, risks, operational performance, and suitability for an intended use.

In short

Model evaluation uses representative data, defined metrics, and human judgment to determine whether a model meets technical and business requirements. It supports model selection and release decisions throughout the foundation model lifecycle, helping engineers balance output quality, safety, latency, cost, and environmental impact.

Cloud Concepts Hub →

6 key aspects of Model Evaluation

Effective evaluation combines representative testing, task-specific metrics, lifecycle controls, business validation, operational comparisons, and responsible resource use.

01
Representative and independent test data

Use held-out examples that reflect production tasks, domains, languages, and difficult cases. Prevent overlap with training or tuning data, and inspect subgroup results so averages do not conceal important weaknesses.

02
Task-specific quality and risk measures

Select metrics that reflect the actual task, such as recall for missed detections or correctness and groundedness for generated answers. Combine automated scoring with calibrated human review, especially where errors have significant consequences.

03
Lifecycle-wide evaluation

Evaluate during model selection, pre-training, fine-tuning, release approval, and production operation. Version evaluation artifacts and rerun regression tests when models, prompts, retrieval data, or application behavior change.

04
Business outcome validation

Connect technical results to measurable outcomes such as handling time, conversion rate, average revenue per user, and customer lifetime value. Use controlled experiments where practical and calculate ROI using total delivery and operating costs.

05
Operational and model-selection trade-offs

Compare suitable model types and capabilities under realistic load, measuring latency, throughput, reliability, and cost per successful task. Treat compliance, supported modalities, context limits, and deployment restrictions as selection constraints rather than optional scoring bonuses.

06
Responsible and sustainable selection

Assess safety, privacy, fairness, and the consequences of model failures alongside output quality. Compare resource consumption for equivalent outcomes and use credible energy or emissions measurements when available rather than assuming that lower price guarantees lower environmental impact.

Cloud services for Model Evaluation

Amazon S3 can hold versioned test datasets and evaluation artifacts, while Amazon Bedrock or Amazon SageMaker AI runs suitable model assessments or custom evaluation workflows. Evaluation results can feed release gates that require minimum quality and safety scores before deployment. Once an application is live, Amazon CloudWatch tracks operational behavior and custom evaluation metrics. Teams can use production failures and appropriately protected user feedback to expand their test sets and rerun comparisons. Together, these services support a repeatable evaluation cycle, but engineers must still define valid metrics, representative data, human review procedures, and business acceptance criteria.

Amazon Bedrock

Amazon Bedrock provides model evaluation capabilities for assessing foundation model outputs through supported automated and human evaluation workflows. Engineers can compare candidate models using relevant datasets and criteria rather than relying only on general benchmark results.

Amazon S3Amazon S3

Amazon S3 can store evaluation datasets, reference answers, generated outputs, and reports. Object versioning and appropriate access controls help teams preserve reproducible evaluation inputs while protecting sensitive test data.

Amazon CloudWatchAmazon CloudWatch

Amazon CloudWatch collects operational metrics and logs used to assess deployed applications, including latency, errors, and custom business or quality measurements. Alarms can flag threshold breaches, but assessing semantic correctness requires an evaluation process that produces those quality measurements.

Model Evaluation exam questions

Real practice exam questions on Model Evaluation, with the correct answer and its explanation. Answer them with every option in the exam simulator.

AIB-C01 (beta)

A retailer uses an Amazon Bedrock foundation model to draft responses to customer complaints. Employees review every draft, correct errors, and save the corrections in the application's feedback database. Monthly reviews reveal that the same types of errors continue to occur. The business sponsor expected the managed model to learn from the saved corrections automatically. No workflow currently uses the feedback to change the application or customize the model. Which recommendation best addresses this situation?

Show answerCORRECT ANSWER

Use the corrections to identify recurring failure patterns, explicitly revise application instructions or pursue supported model customization, and evaluate the changes before release while continuing quality reviews.

A model invoked through Amazon Bedrock does not automatically learn from corrections stored by the application. Monitoring supplies evidence for deliberate improvements. Those improvements require explicit implementation and evaluation, followed by continued monitoring to determine whether they resolve the errors and remain effective.

MLA-C02 (beta)

A healthcare provider is selecting between two foundation models in Amazon Bedrock for a patient-services assistant. The assistant must answer routine questions about appointments and billing, but it must direct requests for clinical advice to a qualified professional without providing that advice. Unnecessarily escalating routine questions would overload the staff.

Both models meet latency and cost requirements and score above 98% on an internal evaluation that consists mostly of routine questions. Before selecting a model, an ML engineer must determine whether each model satisfies both behavioral requirements.

Which evaluation approach should the engineer use?

Show answerCORRECT ANSWER

Evaluate held-out routine questions and clinical-advice requests, including indirect requests, against an expert-approved rubric. Measure inappropriate clinical answers and unnecessary escalations separately, and require both rates to meet predefined limits.

This approach tests both safety and usefulness. Separate error rates prevent success on common routine questions from masking unsafe clinical answers, while also detecting a model that achieves safety by escalating too many routine requests.

SAP-C03

A retail company uses Amazon Bedrock to translate business questions into read-only SQL queries for a reporting application. Its release pipeline evaluates prompt and model changes against 2,000 question-and-reference-result pairs. Developers can inspect all evaluation failures and have repeatedly used these examples to revise prompts. The latest candidate achieves 99% accuracy on this dataset, but a separate pilot reveals frequent errors on previously unseen questions about fiscal periods.

Before promoting the candidate, the company needs evidence that it improves semantic correctness over the production configuration on unseen requests. The evaluation must distinguish a configuration improvement from differences in test difficulty and account for variability in generated responses.

Which evaluation design meets these requirements?

Show answerCORRECT ANSWER

Create a protected, representative holdout dataset that was not used for prompt or model selection, including fiscal-period cases. Evaluate both configurations on the same cases against reference results in an isolated database snapshot, with repeated generations. Apply predefined acceptance criteria and retain the dataset, model, prompt, and inference-setting versions with the results.

A previously unused holdout provides independent evidence of performance beyond examples used during development. Testing both configurations on identical cases and a fixed database snapshot controls differences in difficulty and data state. Reference-result comparisons assess semantic correctness, repeated generations reveal variability, and versioned evidence supports reproducible release decisions.

Practice Model Evaluation in a full exam

Questions on Model Evaluation in each practice exam, with every answer option and an explanation for each:

Start a free practice exam

What is Model Evaluation

3 min read

Model evaluation answers a practical question: is this model suitable for this task under the conditions in which it will operate? A useful evaluation defines the intended users, supported tasks, unacceptable failures, and measurable acceptance criteria before comparing models. For a customer-support assistant, those criteria might include factual correctness, appropriate escalation, protection of customer information, response latency, and cost per successfully resolved case. A strong score on a general benchmark does not establish readiness for a particular business application.

Evaluation occurs throughout the foundation model lifecycle rather than only immediately before deployment. During data selection, engineers check coverage, quality, licensing, and representation of relevant users and domains. During model selection, they compare candidate models against a common baseline. Pre-training and fine-tuning use validation results to guide development, while a separate held-out test set supports final assessment. After deployment, production monitoring and user feedback identify regressions, changing requirements, and new failure cases that inform subsequent evaluation and improvement.

Representative evaluation data should include typical requests, difficult cases, unsupported requests, and adversarial inputs. Keep test examples separate from training and tuning data to reduce leakage and misleading results. Evaluate across relevant domains, languages, user groups, and input lengths rather than reporting only an overall average. For example, a summarization model may perform well on general correspondence but omit critical qualifications in legal documents. Repeated trials can reveal variability in generative outputs; record model versions, prompts, inference settings, and dataset versions so comparisons are reproducible.

Metrics must match the task. Classification can use precision, recall, and F1, while generated answers often require assessments of correctness, relevance, completeness, groundedness, and instruction following. Groundedness measures support from supplied evidence; it does not prove that the evidence itself is correct. Text-overlap metrics such as ROUGE can help compare summaries with references, but they do not establish factual accuracy or usefulness. Automated model-based judges can scale assessment, but their scores can be biased or inconsistent, so calibrate them against human ratings and use expert review for high-impact decisions.

In the cloud, engineers can run repeatable evaluation jobs against hosted model endpoints, store versioned datasets and results, and enforce release thresholds in deployment pipelines. Measure operational behavior under realistic concurrency, including tail latency, throughput, failures, and token consumption. Evaluate the complete application as well as the underlying model: prompts, retrieval, access controls, guardrails, and tools can materially change outcomes. A retrieval failure should not automatically be attributed to the model.

Technical metrics and business value are related but distinct. A support assistant might improve answer accuracy without reducing handling time if employees must heavily review every response. Measure efficiency through time saved or successful tasks per hour, and validate revenue effects through controlled experiments using conversion rate or average revenue per user. Customer lifetime value may require longer observation. Calculate ROI from attributable benefits minus total costs, divided by total costs, including integration, inference, evaluation, monitoring, and human review. A/B tests help distinguish application impact from unrelated business changes.

Model selection therefore involves trade-offs among model type, modality, capabilities, context limits, complexity, quality, latency, cost, and deployment constraints. Compliance requirements may restrict data locations, retention, or permitted uses regardless of benchmark scores. Responsible evaluation also tests harmful outputs, privacy exposure, and performance disparities. Consider sustainability by comparing resource requirements and, where reliable measurements exist, energy use or emissions for equivalent successful workloads. Price alone is not a reliable measure of environmental impact. The best choice is the least resource-intensive option that meets the required quality, safety, and operational thresholds, not automatically the largest or highest-scoring model.

Learn Model Evaluation in lessons

14 short lessons on Model Evaluation in 5 certification courses.

and 8 more lessons

Open the lessons