Population Stability Index

Population Stability Index (PSI) measures distributional change between a reference population and a comparison population. In machine learning operations, the reference might be a model's training dataset, a validation dataset, or a representative production period. The comparison population is typically a recent inference window. Engineers commonly calculate PSI separately for important input features and model prediction scores to detect changes in the population served by a deployed model.

To calculate PSI, define bins using the reference data and apply the same bin definitions to both populations. For each bin, let p be the reference proportion and q be the current proportion. Calculate PSI as the sum of (q − p) × ln(q / p) across all bins, using the natural logarithm. With positive proportions, the result is nonnegative and equals zero when the binned distributions match. Mathematically, this is the Jeffreys divergence, the sum of the two directional Kullback–Leibler divergences, applied to the binned distributions.

For numerical features, reference quantiles are a common starting point, although business-defined intervals can be more interpretable. For categorical features, bins can correspond to individual categories or documented category groups. Preserve the bin boundaries across monitoring windows; recalculating quantiles independently for each window can conceal drift. Define explicit handling for missing values, unseen categories, and values outside the reference range. Zero proportions make logarithms or ratios undefined, so implement a consistent smoothing policy and renormalize proportions when necessary.

For example, a credit-risk model might use ten baseline income bins, each containing approximately 10% of applicants. If a new acquisition channel brings substantially more low-income applicants, their concentration in the lower bins increases PSI. That signal does not establish that the model has become inaccurate. It indicates that the population has changed and prompts checks of channel mix, preprocessing, segment-level outcomes, and eventual labeled performance.

A practical cloud implementation captures a representative sample of inference inputs and scores, stores versioned baseline definitions in object storage, and runs scheduled comparison jobs. Those jobs publish per-feature PSI, observation counts, window timestamps, and baseline versions to a monitoring system. Alerts should account for minimum sample sizes, expected seasonality, and sustained changes. On AWS, custom processing or monitoring code can calculate PSI; engineers should not assume that every built-in data-quality drift statistic is PSI.

Common rules of thumb interpret values below 0.10 as relatively stable, values from 0.10 to 0.25 as requiring investigation, and values above 0.25 as substantial change. These are conventions, not universal decision boundaries, statistical significance levels, or service defaults. Calibrate thresholds against historical variability and operational impact. Sparse bins, small samples, smoothing choices, and the number of bins can materially affect results.

PSI detects changes in individual distributions, not model accuracy or concept drift directly. It can miss changes within bins, changes in relationships between features, and changes in the relationship between inputs and outcomes when input distributions remain stable. Pair it with schema checks, missing-value monitoring, segment analysis, and labeled metrics such as precision, recall, or prediction error. For foundation-model applications, PSI can track structured attributes such as prompt length or evaluation-score distributions, but it does not directly measure semantic drift, factual correctness, or response quality. Treat an alert as an investigation trigger rather than an automatic retraining instruction.