Forum Discussion

binitafulpagare's avatar
binitafulpagare
Icon for Kudo Kingpin rankKudo Kingpin
3 days ago

Scaling Machine Learning Workloads in Microsoft Fabric: How Do You Optimize Performance and Cost?

 

Building a machine learning model on a small dataset is relatively straightforward, but scaling the same workflow to millions of records introduces challenges related to performance, memory, compute resources, and cost.

 

Microsoft Fabric provides Spark notebooks, Lakehouse, pipelines, and MLflow to support machine learning workflows. However, optimizing large-scale ML workloads requires careful architectural decisions.

 

I'd like to hear how the community approaches the following challenges:

 

1. **Large Datasets:** How do you efficiently process millions or billions of records using Fabric Spark notebooks?

2. **Performance Optimization:** When would you use Spark MLlib instead of Python libraries such as Scikit-learn or XGBoost?

3. **Memory Management:** What strategies do you use to avoid out-of-memory errors during feature engineering and model training?

4. **Cost Optimization:** How do you balance Fabric capacity consumption with model training performance?

5. **Workflow Optimization:** How do you reduce unnecessary data processing and avoid repeatedly executing expensive notebook operations?

 

**My key question:** What architecture and optimization techniques would you recommend for building scalable, cost-efficient machine learning pipelines in Microsoft Fabric when working with large datasets?

 

I'd appreciate practical examples, performance optimization tips, and real-world experiences from the community.

data science

3 Replies

  • v-achippa's avatar
    v-achippa
    Icon for Community Support rankCommunity Support
    2 days ago

    Hi binitafulpagare​,

    Thank you for reaching out to Microsoft Fabric Community.

    For large-scale machine learning workloads in Microsoft Fabric, I recommend using PySpark with fabric Lakehouse and Spark notebooks. Please follow below:

    • Use Spark DataFrames for distributed processing instead of loading large datasets into Pandas.
    • Apply filters early and select only the required columns to reduce unnecessary data processing.
    • Use Spark MLlib for distributed machine learning workloads. Use Scikit-learn when the prepared training data and model fits in memory.
    • Avoid unnecessary caching and collecting large datasets to the driver to prevent memory issues.
    • Store prepared data in Delta tables and reuse it instead of repeating expensive transformations.
    • Monitor Spark resource usage and use autoscaling where appropriate to balance performance and cost.

    Use Fabric pipelines to orchestrate data preparation, model training, scoring and MLflow to track experiments and model versions.

    For more details, refer below documentation:
    Train machine learning models in Microsoft Fabric
    Spark monitoring and performance optimization best practices - Microsoft Fabric | Microsoft Learn

    Thanks and regards,
    Anjan Kumar Chippa

    • binitafulpagare's avatar
      binitafulpagare
      Icon for Kudo Kingpin rankKudo Kingpin
      2 days ago

      Hi Anjan,

       

      Thank you for sharing these practical recommendations and the useful documentation links.

       

      I agree that using Spark DataFrames, applying filters early, avoiding unnecessary caching, and reusing prepared Delta tables can significantly improve performance when working with large datasets.

       

      I have one follow-up question:

       

      When training machine learning models on very large and highly imbalanced datasets, how would you handle distributed feature engineering and class imbalance in Microsoft Fabric?

       

      Would you recommend implementing techniques such as class weighting, distributed sampling, or SMOTE before model training?

       

      I'd appreciate your thoughts on the most efficient approach while maintaining model performance and controlling compute costs.

       

      Thanks again for your valuable insights!

      • v-achippa's avatar
        v-achippa
        Icon for Community Support rankCommunity Support
        5 hours ago

        Hi binitafulpagare​,

        For large and highly imbalanced datasets, I would recommend starting with class weighting, if the selected ML algorithm supports it. This helps the model give more importance to minority class records without increasing the dataset size.

        If that does not give good results, consider distributed undersampling of the majority class. SMOTE can also be considered for smaller datasets, but for very large datasets, it can add significant processing and memory overhead.

        Make sure to apply sampling only to the training data and keep the validation and test the data in their original distribution to make sure reliable model evaluation.

        Thanks and regards,
        Anjan Kumar Chippa