Cross-Validation

Cross-validation is a resampling-based approach to estimating performance on unseen data. In standard k-fold cross-validation, engineers divide the development dataset into k nonoverlapping folds, train a fresh model on k minus one folds, and evaluate it on the remaining fold. Each fold serves as validation data once. The resulting scores reveal both typical performance and sensitivity to the particular training and validation examples. Five or ten folds are common starting points, not universal requirements.

Reserve a separate test dataset before using cross-validation for model selection. Compare candidate configurations using their cross-validation results, choose a configuration, and retrain it on the full development dataset before evaluating it on the untouched test set. Repeatedly consulting test results to adjust features, prompts, or hyperparameters turns that test set into another development resource. Nested cross-validation provides an alternative assessment procedure: inner folds select configurations, while outer folds evaluate the complete selection process.

Choose splits according to the data-generating process. Stratified folds help preserve class proportions for classification, although each fold still needs enough minority-class examples for meaningful metrics. Group-aware folds keep related records, such as visits from the same patient or transactions from the same customer, together when the goal is generalization to unseen groups. For forecasting, use chronological or rolling-origin evaluation that trains on earlier observations and validates on later ones. Random folds can otherwise expose future information and produce unrealistic scores.

All learned preprocessing must respect fold boundaries. Fit imputers, scalers, feature selectors, and target encoders only on the training portion of each fold, then apply those fitted transformations to its validation portion. Apply oversampling only within the training portion as well. A practical implementation wraps preprocessing and the estimator in a pipeline that the cross-validation routine fits independently for each fold. Computing normalization statistics or selecting features on the entire dataset before splitting introduces leakage.

For example, an engineer evaluating a rare-event classifier might compare candidate configurations using stratified five-fold cross-validation and precision–recall AUC, then examine recall and precision at an operationally acceptable threshold. Accuracy alone could conceal failure on the rare class. Use the same splits across candidates to make comparisons more meaningful. Record fold-level scores, relevant subgroup results, and aggregate metrics; fold standard deviation describes split sensitivity but is not automatically a statistically valid confidence interval because training sets overlap.

In cloud environments, fold evaluations can run as separate training jobs, or a single training script can execute the entire cross-validation loop. Store dataset versions, split assignments, code versions, random seeds, and metrics so results can be reproduced. Parallel jobs reduce elapsed time but increase concurrent compute demand, storage activity, and quota usage. Evaluating 20 configurations across five folds usually requires 100 fits, before final refitting or any nested evaluation. A tuning service does not necessarily perform cross-validation automatically; its objective must explicitly aggregate the intended fold results.

For generative AI, cross-validation is useful when comparing fine-tuning procedures on limited labeled examples: fine-tune a fresh model for each training fold and evaluate on held-out prompts or tasks. Keep related conversations, document-derived questions, and near-duplicate examples together to avoid leakage. Evaluate task success, factuality, groundedness, safety, and operational measures such as latency and cost where relevant. Retrieval systems should exclude held-out answers or evaluation annotations from development artifacts, while retaining documents that would legitimately be available at inference time.

Evaluating a fixed foundation model across several prompt subsets is useful, but it is not equivalent to retraining across folds. Prompt and retrieval configuration selection can still overfit evaluation data, so preserve independent final assessment data and use human review where automated metrics are insufficient. Cross-validation does not eliminate dataset bias, guarantee future performance under distribution shift, or replace production monitoring. Its main trade-off is stronger evidence for model selection at the cost of repeated computation and more careful experiment design.