Hi @omkar3888 ,
Thank you for reaching out to the Microsoft Community Forum.
If you wrapped your scikit‑learn DecisionTreeClassifier + TfidfVectorizer pipeline in a regular Python UDF (per-row), Spark has to shuttle each row across the JVM <-> Python boundary, which is extremely slow and often collapses execution to effectively one core/node. Regular UDFs operate one value at a time and suffer from serialization overhead; vectorized Pandas UDFs via Arrow process batches, dramatically reducing that overhead.
Fabric’s default/starter pool supports single-node clusters driver and executor co-located. If your workspace/pool/environment is set to minimum nodes = 1, you will observe exactly one node working, regardless of data size. Consider a custom Spark pool or higher node count and appropriate executor/core settings.
If you are collecting to the driver like df.collect().toPandas() then writing or writing with features like Optimize Write and V‑Order without tuning, the write phase can dominate. Optimize Write introduces an extra shuffle to produce larger files; Microsoft’s docs note it can add ~15% to write times on average (in exchange for much faster reads). Misconfigured writes or small-file patterns can explode latency.
Spark best-practices not enabled, If Adaptive Query Execution (AQE), proper partitioning, and column pruning aren’t in place, you can get skewed partitions and inefficient shuffles. Fabric’s Spark basics guide calls these out as default tuning steps.
Please try below things to fix the issue.
1. Use Fabric’s scalable batch scoring instead of manual UDFs. If you registered the model in Fabric (MLflow), call it with the PREDICT path:
MLFlowTransformer API (SynapseML) on a Spark DataFrame cluster-distributed scoring, no per-row Python overhead. Please refer sample code.
from synapse.ml.mlflow import MLFlowTransformer
pred_df = (MLFlowTransformer()
.setModelUri("models:/YourModelName/1")
.transform(input_df))
Note: This approach is built for Fabric and scales out across executors.
2. Check cluster settings,
Workspace --> Spark Compute: node count > 1? autoscale enabled appropriately?
Environment --> Compute: driver/executor cores/memory tuned?
3. Look for task/executor usage, Python UDF time, shuffle hotspots, and write stages.
4. Enable basics, AQE on, DataFrame APIs (not RDD), prune columns before prediction, avoid wide rows.
5. Write path, Ensure you don’t collect() before writing. Use distributed .write.format("delta") and tune Optimize Write/V‑Order as per workload.
Note: Register your model with MLflow in Fabric. Use MLFlowTransformer or predict_batch_udf for distributed scoring. Write predictions directly to Delta; test with Optimize Write on/off and adjust bin size for your latency target. Verify multi-node compute and monitor the run. This typically moves throughput from hundreds of rows/sec (scalar UDF) to tens/hundreds of thousands of rows/sec (batch/Arrow), and ensures all executors participate.
Please refer below links.
Model scoring with PREDICT - Microsoft Fabric | Microsoft Learn
Lakehouse and Delta Tables - Microsoft Fabric | Microsoft Learn
Tutorial: Perform batch scoring and save predictions - Microsoft Fabric | Microsoft Learn
Workspace administration settings in Microsoft Fabric - Microsoft Fabric | Microsoft Learn
Compute Management in Fabric Environments - Microsoft Fabric | Microsoft Learn
Spark Basics - Microsoft Fabric | Microsoft Learn
Tutorial: Train and register machine learning models - Microsoft Fabric | Microsoft Learn
Delta Lake table optimization and V-Order - Microsoft Fabric | Microsoft Learn
Using optimize write on Apache Spark to produce more efficient tables - Azure Synapse Analytics | Mi...
I hope this information helps. Please do let us know if you have any further queries.
Regards,
Dinesh
