Training-serving skew occurs when the inputs used to develop and evaluate a model do not faithfully represent the inputs available during production inference. Differences can involve feature values, schemas, preprocessing logic, freshness, or data distributions. For example, a fraud model might be trained with transaction amounts standardized using one set of statistics while the serving application uses different statistics. The request can satisfy the expected schema and still produce unreliable predictions because the feature values have changed meaning.
A practical prevention strategy is to define a feature contract covering data types, units, missing-value handling, categorical encodings, aggregation windows, and freshness expectations. Reuse versioned transformation code and fitted preprocessing artifacts wherever possible. A scaler or vocabulary learned from the training partition should be persisted and applied during evaluation and inference, rather than independently fitted on each dataset. Package compatible model and preprocessing versions together, and test their behavior on representative records before deployment.
Feature stores help manage consistency between batch training and low-latency inference. An offline store retains historical feature records for dataset construction, while an online store provides recent feature values for prediction requests. In Amazon SageMaker Feature Store, feature groups define record identifiers, event-time fields, and feature schemas. Teams can ingest computed features into the configured stores and retrieve online records during inference. However, storing features centrally does not automatically make the transformations that produce them consistent, and a feature store is not a substitute for transformation versioning or validation.
Historical correctness is particularly important when constructing training datasets. Suppose a credit-risk model predicts whether a customer will default using a rolling payment-history feature. Each training example must use only information available at its prediction time. Joining every historical example to the customer's current feature values can introduce information from the future, creating leakage and unrealistically strong offline results. Point-in-time joins should respect event timestamps and, where records can arrive late or be corrected, actual availability times. Historical storage alone does not guarantee that a dataset satisfies these requirements.
Batch and streaming pipelines also need equivalent semantics. A nightly job might count purchases over complete calendar days, while an online process counts purchases over the preceding 24 hours. Both outputs may be named purchase_count, but they represent different features. Define window boundaries, time zones, deduplication rules, and handling of late events explicitly. Monitor feature age as well as values, because a correctly calculated but stale feature can still violate the model's serving assumptions.
Consistency has trade-offs. Precomputing features reduces inference latency but introduces storage costs and potential staleness. Computing features during each request improves access to current information but adds latency and dependency failures. Shared code reduces implementation differences, although different batch and online execution engines still require parity testing. Useful tests replay identical raw inputs through both paths and compare the resulting feature vectors within appropriate numerical tolerances.
Training-serving skew is not simply another name for model drift. Natural changes in production data can create a training-serving distribution mismatch, but inconsistent preprocessing can cause skew even when the underlying population is unchanged. Distribution monitoring can reveal symptoms, while matched-record comparisons help identify transformation defects. Retraining alone will not fix a serving pipeline that applies the wrong encoding, retrieves stale records, or supplies features in the wrong order.
AWS Glue