How Do You Validate ML Models for Real-World Performance?

Hi binitafulpagare​,

For time-dependent ML problems, I would normally move beyond a single train/test split and use rolling or expanding-window validation.

For example, with an expanding window you might train on Jan-Mar and validate on Apr, then train on Jan-Apr and validate on May, then Jan-May and validate on Jun. A sliding window is similar, except older observations eventually fall out of the training set.

I prefer this over random cross-validation when the production model will always predict the future from the past, because each validation fold preserves that ordering and gives you a much better view of how performance changes as the data distribution drifts.

For imbalanced classification, I would also keep the temporal split intact and apply any resampling, feature scaling, encoding, or threshold tuning only inside the training portion of each fold. Otherwise information from the validation period can leak back into the training process.

In Fabric, I would log each fold as part of an MLflow experiment rather than only logging the final model. Microsoft's ML experiment guidance supports tracking parameters, metrics, tags, code versions, and outputs for each run, so you can compare not only the average score but also how stable the model is across different time periods.

With the newer MLflow 3 support in Fabric, you can also tie logged models back to the runs, parameters, metrics, code, and datasets that produced them, which is useful when trying to reproduce why one validation window behaved differently from another.

For concept drift, I would monitor both model metrics and feature distributions over time. If the model performs well on older folds but degrades consistently on recent ones, I would treat that as a signal that the underlying relationship may be changing rather than simply retraining blindly.

So my usual pattern would be:

• time-ordered rolling or expanding validation
• all preprocessing fitted only on the training fold
• imbalance handling inside each training fold
• MLflow tracking for every fold and model version
• a final untouched holdout period representing the most recent production-like data

That gives you a much more realistic estimate of how the model is likely to behave after deployment than standard random cross-validation.

AI-assisted drafting: AI was used to help structure and phrase this response. I reviewed and validated the technical content before posting.

Hi binitafulpagare​ 

Data leakage is one of the common reasons why ML models perform well during validation but fail to deliver similar results in production.

Selecting the appropriate validation strategy is essential. For time-dependent datasets, time-based splitting generally provides a more realistic evaluation than random splitting. For imbalanced classification, stratified sampling helps maintain class distribution across training and validation datasets.

Additionally, ensure that the entire feature engineering and preprocessing pipeline is validated to prevent future information from leaking into training data.

In Microsoft Fabric, MLflow can help track experiment parameters, evaluation metrics and model versions, making validation runs more reproducible and easier to compare.

For more details, refer to:

• Machine Learning Experiments in Microsoft Fabric
• Train and Register ML Models in Fabric

It would also be helpful to understand how others are implementing these validation strategies in their Microsoft Fabric environments.