Random Search

Random Search is a method for finding useful hyperparameter settings without exhaustively evaluating every possible combination. Hyperparameters are settings supplied to a training process, such as learning rate, tree depth, regularization strength, and batch size. They differ from model parameters, such as neural network weights, which the training algorithm learns from data. Each sampled hyperparameter configuration defines a trial that trains and evaluates a model.

The process starts by defining a search space and an objective metric. Search spaces can contain continuous ranges, integer ranges, and categorical choices. The objective must specify whether higher or lower values are better: validation accuracy is typically maximized, whereas validation loss is minimized. Use a consistent validation procedure across trials, and reserve a separate test dataset for final evaluation rather than repeatedly using it to select configurations.

Sampling distributions strongly influence the usefulness of the search. For example, a learning rate between 0.00001 and 0.1 often benefits from logarithmic sampling, which spreads trials across orders of magnitude. Linear sampling across that same interval places most samples toward the larger orders of magnitude. Categorical settings, such as an optimizer choice, require explicit candidate values. Search-space boundaries should exclude invalid settings and configurations that cannot fit available memory.

Consider tuning a classifier with learning rate, maximum tree depth, and regularization strength. Instead of evaluating a fixed grid of every combination, an engineer might allow 40 randomly sampled trials, with five running concurrently. The tuning system launches training jobs, records a validation metric for each, and identifies the best observed configuration. The trial limit controls the amount of exploration, while the concurrency limit controls how quickly resources are consumed. Higher concurrency can reduce elapsed time but does not inherently reduce total compute cost.

Cloud platforms apply this pattern by coordinating training jobs, provisioning compute, collecting metrics, and storing model artifacts. In Amazon SageMaker AI automatic model tuning, select the Random tuning strategy, define tunable hyperparameter ranges, configure the objective metric, and set total-job and parallel-job limits. Custom training code must expose the objective metric in a form the tuning service can collect, commonly through training logs and configured metric definitions. A successful training job is not sufficient if its objective metric is missing or incorrectly parsed.

Unlike Bayesian optimization, basic Random Search does not use completed trial results to decide where to sample next. This makes independent parallel trials straightforward and avoids waiting for earlier results before generating configurations. Unlike grid search, it does not systematically enumerate combinations. Random Search can explore more distinct values of influential hyperparameters under a limited budget, particularly when many other hyperparameters have little effect. It does not guarantee the global optimum or uniformly cover every region of the search space.

Random Search also differs from Hyperband, which allocates training resources across configurations and eliminates weaker candidates at intermediate stages. Random sampling and early stopping are separate mechanisms; a random strategy does not inherently stop unpromising trials. When comparing methods, consider both how configurations are selected and how much training each receives.

For foundation model fine-tuning, the same approach can explore supported settings such as learning rate, batch size, and epoch count, subject to the training interface's capabilities. Large-model trials can be expensive, so start with defensible ranges and a bounded budget. Record sampled configurations, data versions, training environments, and random seeds. A sampling seed improves repeatability of configuration selection, but training can still vary because of stochastic optimization or nondeterministic hardware operations. Re-evaluate promising configurations when small metric differences could reflect noise rather than meaningful improvements.