Forum Discussion
Kudo KingpinHow Do You Validate ML Models for Real-World Performance?
Community SupportData leakage is one of the common reasons why ML models perform well during validation but fail to deliver similar results in production.
Selecting the appropriate validation strategy is essential. For time-dependent datasets, time-based splitting generally provides a more realistic evaluation than random splitting. For imbalanced classification, stratified sampling helps maintain class distribution across training and validation datasets.
Additionally, ensure that the entire feature engineering and preprocessing pipeline is validated to prevent future information from leaking into training data.
In Microsoft Fabric, MLflow can help track experiment parameters, evaluation metrics and model versions, making validation runs more reproducible and easier to compare.
For more details, refer to:
It would also be helpful to understand how others are implementing these validation strategies in their Microsoft Fabric environments.
Kudo KingpinHi v-aatheeque,
Thank you for sharing these valuable insights and the documentation references.
I agree that time-based splitting is essential for time-dependent datasets, while stratified sampling helps maintain class distribution in imbalanced classification problems.
I have one follow-up question:
For time-dependent machine learning problems, how would you handle situations where the data distribution changes over time?
Would you recommend using rolling-window cross-validation, expanding-window validation, or walk-forward validation to evaluate model stability before production deployment?
Also, how would you integrate this validation process with MLflow experiment tracking in Microsoft Fabric?
I'd appreciate your recommendations and practical implementation insights.
Thanks again!