Classification Metrics

Classification models predict discrete categories, such as fraudulent versus legitimate transactions or the department responsible for a support ticket. Classification metrics compare these predictions with ground-truth labels. During the ML development lifecycle, engineers use them to compare candidate models, tune decision thresholds, approve deployments, and monitor production quality. No single metric captures every consequence of a prediction.

For binary classification, first define which outcome is positive. A true positive (TP) is a correctly identified positive case, while a true negative (TN) is a correctly identified negative case. A false positive (FP) incorrectly flags a negative case, and a false negative (FN) misses a positive case. A confusion matrix summarizes these counts. In fraud detection, for example, a false positive can block a legitimate purchase, while a false negative can allow financial loss.

Accuracy is (TP + TN) / (TP + TN + FP + FN): the fraction of all predictions that are correct. It is useful when classes are reasonably balanced and errors have similar consequences, but it can conceal poor minority-class performance. If only 1% of transactions are fraudulent, a model that always predicts legitimate achieves 99% accuracy while detecting no fraud. High accuracy alone therefore does not establish that a classifier is useful.

Precision is TP / (TP + FP): among predicted positives, how many are actually positive? Recall is TP / (TP + FN): among actual positives, how many did the model identify? Favor precision when false alarms are especially costly, such as automatically blocking legitimate customers. Favor recall when missing a positive case is especially costly, such as screening for a serious condition. These priorities still require business-specific constraints rather than a universal choice.

F1 score is the harmonic mean of precision and recall: 2 × precision × recall / (precision + recall). It is useful when both matter, but it does not account for true negatives or explicitly represent financial error costs. Consider 10,000 transactions containing 100 fraud cases. A classifier that identifies 80 fraud cases and incorrectly flags 40 legitimate transactions has 66.7% precision, 80% recall, approximately 72.7% F1, and 99.4% accuracy. These values describe very different aspects of the same model. Evaluation code should explicitly handle metrics whose denominators are zero.

Many classifiers produce scores that become labels through a decision threshold. Lowering the threshold generally increases recall while admitting more false positives; raising it generally reduces false positives while missing more positives. Select thresholds using validation data and operational costs, then evaluate the chosen configuration on a separate test set. For multiclass tasks, report per-class results and specify the averaging method: macro averaging gives each class equal weight, while support-weighted averaging gives more influence to frequent classes.

In cloud deployments, evaluation jobs can read versioned datasets from object storage, compute metrics, and record results alongside model versions. Use representative, leakage-free data and examine important customer segments rather than only aggregate scores. After deployment, join captured predictions with subsequently obtained ground-truth labels to measure actual performance. Input drift can signal a problem, but it does not by itself prove that accuracy has decreased.

Technical evaluation must be paired with business evaluation. Track cost per user using a defined cost boundary, including relevant inference and operating expenses, and account for development, labeling, training, and maintenance costs. Measure customer feedback, support burden, and downstream outcomes through controlled rollouts where appropriate. ROI can be expressed as net benefit divided by investment over a defined period. A higher F1 score does not guarantee higher ROI if it requires expensive infrastructure, creates more manual reviews, or worsens the customer experience.