Forum Discussion

Kudo Kingpin

3 days ago

Scaling Machine Learning Workloads in Microsoft Fabric: How Do You Optimize Performance and Cost?

Building a machine learning model on a small dataset is relatively straightforward, but scaling the same workflow to millions of records introduces challenges related to performance, memory, compu...

data science

v-achippa

Community Support

2 days ago

Hi binitafulpagare,

Thank you for reaching out to Microsoft Fabric Community.

For large-scale machine learning workloads in Microsoft Fabric, I recommend using PySpark with fabric Lakehouse and Spark notebooks. Please follow below:

Use Spark DataFrames for distributed processing instead of loading large datasets into Pandas.
Apply filters early and select only the required columns to reduce unnecessary data processing.
Use Spark MLlib for distributed machine learning workloads. Use Scikit-learn when the prepared training data and model fits in memory.
Avoid unnecessary caching and collecting large datasets to the driver to prevent memory issues.
Store prepared data in Delta tables and reuse it instead of repeating expensive transformations.
Monitor Spark resource usage and use autoscaling where appropriate to balance performance and cost.

Use Fabric pipelines to orchestrate data preparation, model training, scoring and MLflow to track experiments and model versions.

For more details, refer below documentation:
Train machine learning models in Microsoft Fabric
Spark monitoring and performance optimization best practices - Microsoft Fabric | Microsoft Learn

Thanks and regards,
Anjan Kumar Chippa

binitafulpagare
Kudo Kingpin
2 days ago
Hi Anjan,

Thank you for sharing these practical recommendations and the useful documentation links.

I agree that using Spark DataFrames, applying filters early, avoiding unnecessary caching, and reusing prepared Delta tables can significantly improve performance when working with large datasets.

I have one follow-up question:

When training machine learning models on very large and highly imbalanced datasets, how would you handle distributed feature engineering and class imbalance in Microsoft Fabric?

Would you recommend implementing techniques such as class weighting, distributed sampling, or SMOTE before model training?

I'd appreciate your thoughts on the most efficient approach while maintaining model performance and controlling compute costs.

Thanks again for your valuable insights!
- v-achippa
  Community Support
  7 hours ago
  Hi binitafulpagare,
  
  For large and highly imbalanced datasets, I would recommend starting with class weighting, if the selected ML algorithm supports it. This helps the model give more importance to minority class records without increasing the dataset size.
  
  If that does not give good results, consider distributed undersampling of the majority class. SMOTE can also be considered for smaller datasets, but for very large datasets, it can add significant processing and memory overhead.
  
  Make sure to apply sampling only to the training data and keep the validation and test the data in their original distribution to make sure reliable model evaluation.
  
  Thanks and regards,
  Anjan Kumar Chippa