Predicted results ingestion takes a lot of time

omkar3888 — Wed, 26 Nov 2025 08:28:00 GMT

Hello Community Users,

i was trying to develope a ML model in fabric using pyspark notebook. i has also registered model decisiontreeclassifier with tfidvectorizer . but after using above registered model for prediction the ingestion of prediction takes hell lot of time. to ingest 10K records it tool 25 min and my prediction dataframe has 3 mn records. I followed each and every step mentioned in fabric github repo. since the model is not working across the entire cluster and i am only getting a single node to work on the process. can somebody explain why this is happening..? am i missing any step.

Re: Predicted results ingestion takes a lot of time

v-dineshya — Thu, 27 Nov 2025 06:27:41 GMT

Hi omkar3888 ,

Thank you for reaching out to the Microsoft Community Forum.

If you wrapped your scikit‑learn DecisionTreeClassifier + TfidfVectorizer pipeline in a regular Python UDF (per-row), Spark has to shuttle each row across the JVM <-> Python boundary, which is extremely slow and often collapses execution to effectively one core/node. Regular UDFs operate one value at a time and suffer from serialization overhead; vectorized Pandas UDFs via Arrow process batches, dramatically reducing that overhead.

Fabric’s default/starter pool supports single-node clusters driver and executor co-located. If your workspace/pool/environment is set to minimum nodes = 1, you will observe exactly one node working, regardless of data size. Consider a custom Spark pool or higher node count and appropriate executor/core settings.

If you are collecting to the driver like df.collect().toPandas() then writing or writing with features like Optimize Write and V‑Order without tuning, the write phase can dominate. Optimize Write introduces an extra shuffle to produce larger files; Microsoft’s docs note it can add ~15% to write times on average (in exchange for much faster reads). Misconfigured writes or small-file patterns can explode latency.

Spark best-practices not enabled, If Adaptive Query Execution (AQE), proper partitioning, and column pruning aren’t in place, you can get skewed partitions and inefficient shuffles. Fabric’s Spark basics guide calls these out as default tuning steps.

Please try below things to fix the issue.

1. Use Fabric’s scalable batch scoring instead of manual UDFs. If you registered the model in Fabric (MLflow), call it with the PREDICT path:

MLFlowTransformer API (SynapseML) on a Spark DataFrame cluster-distributed scoring, no per-row Python overhead. Please refer sample code.

from synapse.ml.mlflow import MLFlowTransformer

pred_df = (MLFlowTransformer()
.setModelUri("models:/YourModelName/1")
.transform(input_df))

Note: This approach is built for Fabric and scales out across executors.

2. Check cluster settings,

Workspace --> Spark Compute: node count > 1? autoscale enabled appropriately?
Environment --> Compute: driver/executor cores/memory tuned?

3. Look for task/executor usage, Python UDF time, shuffle hotspots, and write stages.

4. Enable basics, AQE on, DataFrame APIs (not RDD), prune columns before prediction, avoid wide rows.

5. Write path, Ensure you don’t collect() before writing. Use distributed .write.format("delta") and tune Optimize Write/V‑Order as per workload.

Note: Register your model with MLflow in Fabric. Use MLFlowTransformer or predict_batch_udf for distributed scoring. Write predictions directly to Delta; test with Optimize Write on/off and adjust bin size for your latency target. Verify multi-node compute and monitor the run. This typically moves throughput from hundreds of rows/sec (scalar UDF) to tens/hundreds of thousands of rows/sec (batch/Arrow), and ensures all executors participate.

Please refer below links.

Model scoring with PREDICT - Microsoft Fabric | Microsoft Learn

Lakehouse and Delta Tables - Microsoft Fabric | Microsoft Learn

Tutorial: Perform batch scoring and save predictions - Microsoft Fabric | Microsoft Learn

Workspace administration settings in Microsoft Fabric - Microsoft Fabric | Microsoft Learn

Compute Management in Fabric Environments - Microsoft Fabric | Microsoft Learn

Spark Basics - Microsoft Fabric | Microsoft Learn

Tutorial: Train and register machine learning models - Microsoft Fabric | Microsoft Learn

Delta Lake table optimization and V-Order - Microsoft Fabric | Microsoft Learn

Using optimize write on Apache Spark to produce more efficient tables - Azure Synapse Analytics | Microsoft Learn

I hope this information helps. Please do let us know if you have any further queries.

Regards,

Dinesh

Re: Predicted results ingestion takes a lot of time

v-dineshya — Mon, 01 Dec 2025 04:23:14 GMT

Hi omkar3888 ,

We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. And, if you have any further query do let us know.

Regards,

Dinesh

Re: Predicted results ingestion takes a lot of time

v-dineshya — Thu, 04 Dec 2025 05:00:05 GMT

Hi @omkar3888 ,

We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. And, if you have any further query do let us know.

Regards,

Dinesh

topic Re: Predicted results ingestion takes a lot of time in Data Science

Predicted results ingestion takes a lot of time

Re: Predicted results ingestion takes a lot of time

Re: Predicted results ingestion takes a lot of time

Re: Predicted results ingestion takes a lot of time