Forum Discussion

Kudo Kingpin

3 days ago

Scaling Machine Learning Workloads in Microsoft Fabric: How Do You Optimize Performance and Cost?

Building a machine learning model on a small dataset is relatively straightforward, but scaling the same workflow to millions of records introduces challenges related to performance, memory, compu...

data science

binitafulpagare

Kudo Kingpin

2 days ago

Hi Anjan,

Thank you for sharing these practical recommendations and the useful documentation links.

I agree that using Spark DataFrames, applying filters early, avoiding unnecessary caching, and reusing prepared Delta tables can significantly improve performance when working with large datasets.

I have one follow-up question:

When training machine learning models on very large and highly imbalanced datasets, how would you handle distributed feature engineering and class imbalance in Microsoft Fabric?

Would you recommend implementing techniques such as class weighting, distributed sampling, or SMOTE before model training?

I'd appreciate your thoughts on the most efficient approach while maintaining model performance and controlling compute costs.

Thanks again for your valuable insights!

v-achippa

Community Support

7 hours ago

Hi binitafulpagare,

For large and highly imbalanced datasets, I would recommend starting with class weighting, if the selected ML algorithm supports it. This helps the model give more importance to minority class records without increasing the dataset size.

If that does not give good results, consider distributed undersampling of the majority class. SMOTE can also be considered for smaller datasets, but for very large datasets, it can add significant processing and memory overhead.

Make sure to apply sampling only to the training data and keep the validation and test the data in their original distribution to make sure reliable model evaluation.

Thanks and regards,
Anjan Kumar Chippa