How Do You Validate ML Models for Real-World Performance?

Hi binitafulpagare​,

For time-dependent ML problems, I would normally move beyond a single train/test split and use rolling or expanding-window validation.

For example, with an expanding window you might train on Jan-Mar and validate on Apr, then train on Jan-Apr and validate on May, then Jan-May and validate on Jun. A sliding window is similar, except older observations eventually fall out of the training set.

I prefer this over random cross-validation when the production model will always predict the future from the past, because each validation fold preserves that ordering and gives you a much better view of how performance changes as the data distribution drifts.

For imbalanced classification, I would also keep the temporal split intact and apply any resampling, feature scaling, encoding, or threshold tuning only inside the training portion of each fold. Otherwise information from the validation period can leak back into the training process.

In Fabric, I would log each fold as part of an MLflow experiment rather than only logging the final model. Microsoft's ML experiment guidance supports tracking parameters, metrics, tags, code versions, and outputs for each run, so you can compare not only the average score but also how stable the model is across different time periods.

With the newer MLflow 3 support in Fabric, you can also tie logged models back to the runs, parameters, metrics, code, and datasets that produced them, which is useful when trying to reproduce why one validation window behaved differently from another.

For concept drift, I would monitor both model metrics and feature distributions over time. If the model performs well on older folds but degrades consistently on recent ones, I would treat that as a signal that the underlying relationship may be changing rather than simply retraining blindly.

So my usual pattern would be:

• time-ordered rolling or expanding validation
• all preprocessing fitted only on the training fold
• imbalance handling inside each training fold
• MLflow tracking for every fold and model version
• a final untouched holdout period representing the most recent production-like data

That gives you a much more realistic estimate of how the model is likely to behave after deployment than standard random cross-validation.

AI-assisted drafting: AI was used to help structure and phrase this response. I reviewed and validated the technical content before posting.