Scaling Machine Learning Workloads in Microsoft Fabric: How Do You Optimize Performance and Cost?

Hi binitafulpagare​,

For very large and highly imbalanced datasets, I would normally start with class weighting rather than oversampling.

Class weighting keeps the original dataset size unchanged while giving the minority class more influence during training. That is usually much cheaper in Spark than generating synthetic observations across a very large dataset.

If the majority class is extremely large, I would next consider distributed undersampling of that class. The important part is to do it reproducibly and only on the training data, while leaving validation and test data in their original distribution.

I would be more cautious with SMOTE at this scale. It can work well for smaller datasets, but creating synthetic observations across millions or billions of records can add significant shuffle, memory and storage overhead. It can also make the training distribution increasingly different from what the model will actually see in production.

For Fabric Spark specifically, Microsoft's Spark performance guidance recommends watching partitioning, skew and executor memory closely. The Spark Advisor can also identify skew and caching problems directly from notebook execution.

I would therefore use a pattern such as:

original Delta training data

-> time or stratified train/validation split

-> class weighting or majority-class undersampling on the training partition only

-> distributed training

-> evaluation against the untouched validation/test distribution

For cost, I would also monitor whether the training job is CPU-bound, memory-bound or simply scanning too much data before increasing capacity. Fabric now supports autoscaling and on-demand Spark billing, so bursty ML training workloads can be isolated from the main Fabric capacity rather than permanently sizing the whole capacity for occasional model-training peaks. Microsoft's on-demand Spark billing guidance is useful for that scenario.

I would also avoid repeatedly rebuilding the same feature dataset for every experiment. Persist the curated feature set as Delta and reuse it across MLflow runs, then vary the model parameters and sampling strategy rather than rerunning the entire feature-engineering pipeline each time.

So for a very large imbalanced dataset, my order would usually be:

• class weighting first
• distributed undersampling if the majority class is still unnecessarily large
• SMOTE only when the dataset is small enough that the extra synthetic data is justified
• and always evaluate against the original production-like class distribution

That tends to give a better balance between model quality, training time and Fabric compute consumption.

AI-assisted drafting: AI was used to help structure and phrase this response. I reviewed and validated the technical content before posting.

Hi binitafulpagare​,

Thank you for reaching out to Microsoft Fabric Community.

For large-scale machine learning workloads in Microsoft Fabric, I recommend using PySpark with fabric Lakehouse and Spark notebooks. Please follow below:

• Use Spark DataFrames for distributed processing instead of loading large datasets into Pandas.
• Apply filters early and select only the required columns to reduce unnecessary data processing.
• Use Spark MLlib for distributed machine learning workloads. Use Scikit-learn when the prepared training data and model fits in memory.
• Avoid unnecessary caching and collecting large datasets to the driver to prevent memory issues.
• Store prepared data in Delta tables and reuse it instead of repeating expensive transformations.
• Monitor Spark resource usage and use autoscaling where appropriate to balance performance and cost.

Use Fabric pipelines to orchestrate data preparation, model training, scoring and MLflow to track experiments and model versions.

For more details, refer below documentation:
Train machine learning models in Microsoft Fabric
Spark monitoring and performance optimization best practices - Microsoft Fabric | Microsoft Learn

Thanks and regards,
Anjan Kumar Chippa