Cloud concept · Updated

Experiment Tracking - Experiment tracking is the systematic recording of machine learning runs, their inputs, configurations, outputs, and evaluation results so teams can compare and reproduce their work.

In short

Experiment tracking connects each model result to the data, code, parameters, and execution environment that produced it. It supports repeatable development, informed model selection, and traceability as models move toward production and are later retrained.

Cloud Concepts Hub →

6 key aspects of Experiment Tracking

Effective experiment tracking combines structured records, reproducible inputs, fair comparisons, automated capture, lifecycle links, and governed storage.

01
Structured run records

Assign each execution a unique identifier and record its configuration, status, metrics, and artifacts. Use consistent naming and tags so engineers can find related work across projects and pipelines.

02
Data and code provenance

Link results to source-code revisions, dataset versions, preprocessing logic, and execution environments. Prefer immutable references over mutable paths when reproducibility matters.

03
Meaningful comparison

Compare candidates using consistent evaluation procedures and relevant metrics, including latency or cost when appropriate. Record dataset splits and decision thresholds to avoid misleading comparisons.

04
Repeatable instrumentation

Build tracking into reusable training code and automated workflows rather than relying on manual notes. Capture failed and interrupted runs as well as successful ones to support debugging and reduce duplicated effort.

05
Lifecycle traceability

Connect selected runs to registered models and deployment records. Use those links to investigate production issues and compare retrained candidates with established baselines.

06
Governed retention and access

Protect experiment metadata and artifacts with appropriate access controls and encryption. Define retention policies and prevent credentials, sensitive samples, or unnecessary personal information from entering logs.

Cloud services for Experiment Tracking

On AWS, SageMaker AI can provide the experiment-tracking layer while Amazon S3 stores datasets and larger artifacts. Training code records parameters and metrics, and each run references the exact data and artifacts used or produced. Azure Machine Learning and Vertex AI offer comparable tracking capabilities within their respective cloud environments; using all three providers is not required. In any implementation, connect tracked runs to workflow execution and model lifecycle records, and keep production monitoring separate but linked so retraining decisions retain a clear history.

Amazon SageMakerAmazon SageMaker

Amazon SageMaker AI offers managed MLflow capabilities for recording and comparing experiment parameters, metrics, and artifacts. Teams can connect tracked work with training workflows and model lifecycle processes, while still defining their own evaluation and promotion criteria.

Amazon S3Amazon S3

Amazon S3 can store training datasets, model artifacts, and evaluation reports referenced by experiment records. Object versioning helps preserve identifiable versions, but S3 alone does not provide experiment comparison or structured run tracking.

Experiment Tracking exam questions

Real practice exam questions on Experiment Tracking, with the correct answer and its explanation. Answer them with every option in the exam simulator.

AIB-C01 (beta)

An insurer is evaluating an AI application that extracts information from policy documents. Before a controlled launch, a single candidate must demonstrate at least 96% field accuracy, a total processing cost of no more than $0.12 per document, and a response time of no more than 2 seconds. These requirements are mandatory.

The team's experiment records link each evaluated configuration to the following results from the same representative test set. Costs include human review.

Run Field accuracy Cost per document Response time
R1 98% $0.16 1.2 seconds
R2 94% $0.08 1.4 seconds
R3 97% $0.09 3.1 seconds

An executive summary highlights the highest accuracy, lowest cost, and fastest response time achieved during experimentation. The sponsor asks whether these results justify launching the application. Which recommendation should the AI business strategist make?

Show answerCORRECT ANSWER

Continue experimentation and assess each candidate's linked results together; advance only a configuration that demonstrates compliance with all three requirements.

Each run represents a particular configuration and its measured outcomes. No recorded run meets every requirement: R1 fails cost, R2 fails accuracy, and R3 fails response time. Using the linked evidence prevents combining strengths from different runs into an unsupported claim of launch readiness.

MLA-C02 (beta)

A team uses a custom launcher to start Amazon SageMaker AI training and evaluation jobs from a shared Git repository. Each job clones the configured Git reference when its compute instance starts. Jobs can remain queued while developers merge additional changes into the repository. MLflow records each job's metrics, dataset reference, container image digest, and Git branch name.

An engineer discovers that two jobs submitted with the same branch name used different preprocessing and evaluation code. The team must ensure that a submitted experiment uses the intended source revision, even if the branch changes before the job starts, and that the source can be retrieved for a later rerun.

Which change meets these requirements?

Show answerCORRECT ANSWER

Commit and push all intended source changes before submission, configure the launcher to check out the full commit SHA, and record that SHA in MLflow while retaining the commit in the repository.

A full commit SHA identifies a specific source snapshot. Committing and pushing the changes makes that snapshot available to the job, checking out the SHA prevents later branch updates from changing the experiment, and retaining the commit and recording its SHA supports later retrieval and reruns.

Practice Experiment Tracking in a full exam

Questions on Experiment Tracking in each practice exam, with every answer option and an explanation for each:

Start a free practice exam

What is Experiment Tracking

3 min read

Machine learning development involves repeated experiments rather than a single build. Engineers change training data, preprocessing, algorithms, or hyperparameters and evaluate the resulting models. Experiment tracking preserves a structured record of these attempts so the team can explain why a model performed differently and identify which configuration should advance. An experiment usually groups related work, while a run represents one execution, such as a training job or an evaluation.

A useful run record includes a unique identifier, purpose, source-code revision, dataset version or immutable reference, preprocessing configuration, hyperparameters, dependency versions, and execution status. It also stores evaluation metrics and links to artifacts such as model files, plots, and evaluation reports. Parameters describe configuration, such as learning rate; metrics describe measured outcomes, such as validation loss. Recording only the final accuracy leaves too little information to reproduce or interpret a result.

In the cloud, training jobs commonly run on temporary compute resources. A tracking service provides persistent metadata storage after those resources terminate, while object storage holds larger artifacts. Training code or an integrated SDK sends parameters and metrics to the tracker. Teams can instrument reusable training templates and pipelines so local development, scheduled training, and distributed jobs follow consistent recording conventions. Unique run identifiers prevent parallel executions from overwriting one another.

For example, a team developing a customer-churn classifier might compare several learning rates using the same versioned dataset and validation split. Tracking precision, recall, training duration, and estimated execution cost helps engineers compare practical trade-offs rather than selecting solely on accuracy. The team should record the decision threshold because precision and recall depend on it. Validation results guide selection; an appropriately protected test set provides a final assessment rather than becoming another repeatedly optimized target.

Tracking supports reproducibility but does not guarantee identical results. Random seeds, library versions, container images, hardware differences, and nondeterministic operations can affect outcomes. A dataset path is also insufficient if its contents can change. Versioned objects, snapshots, or content hashes provide stronger references. These practices reduce technical debt by replacing undocumented notebook state and informal filenames with records that other engineers can understand and reuse.

Experiment tracking is distinct from hyperparameter optimization, workflow orchestration, model registration, and production monitoring. A tuner chooses configurations, an orchestrator coordinates execution, a registry manages model versions and lifecycle metadata, and monitoring observes deployed behavior. Tracking records evidence from development and evaluation. It does not automatically select a safe model, approve deployment, detect production drift, or initiate retraining.

Within MLOps, teams connect a selected run to its registered model and deployment. If production monitoring detects degraded performance or changing data, a retraining workflow can create new tracked runs and compare candidate models against the current baseline. Production readiness still requires suitable validation, security checks, deployment testing, and approval controls. Retaining failed runs and rejected candidates can preserve useful knowledge, but artifact storage, metadata volume, and instrumentation add cost. Apply access controls, avoid logging secrets or unnecessary personal data, and define retention policies that preserve essential evidence without keeping every intermediate artifact indefinitely.

Learn Experiment Tracking in lessons

5 short lessons on Experiment Tracking in 4 certification courses.

Open the lessons