Forum Discussion
Fabric Pipeline Performance
Hi udhaya208,
Thank you for the update, and it is great to hear that consolidating the notebooks helped reduce the execution time from 17 minutes to around 10 minutes. This improvement confirms that a significant portion of the earlier delay was related to pipeline orchestration and repeated Spark session initialization overhead.
Since the execution time is still higher compared to Talend, the remaining bottleneck is now likely within the Spark processing layer itself rather than the pipeline orchestration. At this stage, we would recommend reviewing the notebook transformations for expensive operations such as large shuffle operations, repeated joins, repartitioning, unnecessary caching, or multiple actions triggering recomputation. Additionally, please verify whether the workload is experiencing capacity throttling or resource contention in the Fabric workspace. You may also see further improvements by optimizing partitioning strategy, enabling adaptive query execution, reducing small file operations, and reviewing the Spark UI/execution metrics to identify slow stages or skewed tasks.Hope this helps. If you have any questions regarding this, please feel free to reach out us. We will be happy to help.
Community SupportHi udhaya208,
Thank you for reaching out to the Microsoft Fabric Community Forum.
The notebooks themselves do not appear to be the primary bottleneck, since they complete in less than a minute when executed independently. The increased runtime is most likely caused by pipeline orchestration overhead and Spark session initialization when the notebooks are triggered sequentially through multiple pipelines. In the current design, each notebook execution may be creating a separate Spark session, which can add several minutes of startup latency and significantly increase the total pipeline execution time.
To improve performance, we recommend reducing nested pipeline execution where possible and running notebooks directly within a single pipeline. Additionally, if there are no dependencies between the notebooks, consider running them in parallel instead of sequentially to reduce cumulative delays. Enabling High Concurrency mode can also help by allowing notebook activities to share the same Spark session and minimize repeated startup overhead. Please review the following Microsoft documentation for detailed guidance on optimizing notebook execution and Spark session reuse in Fabric pipelines.
Please refer to the below links:
1. Configure high concurrency mode for notebooks in pipelines - Microsoft Fabric | Microsoft Learn
2. High concurrency mode in Apache Spark compute for Fabric - Microsoft Fabric | Microsoft Learn
3. Notebook activity - Microsoft Fabric | Microsoft Learn
Hope this helps. If you have any questions regarding this, please feel free to reach out us. We will be happy to help.
- v-kpoloju-msft
Hi udhaya208,
Just checking in to see if the issue has been resolved on your end. If the earlier suggestions helped, that’s great to hear! And if you’re still facing challenges, feel free to share more details happy to assist further.Thank you.
Hi v-kpoloju-msft,
Thanks for your suggestion. I really appreciate your Efforts. As recommended, I removed the sequential pipeline execution and consolidated the four notebooks into a single notebook. This optimization reduced the execution time from 17 minutes to 10 minutes for the same volume of records. However, our expectation is to bring it down to less than 5 minutes.
For some background, we are currently migrating the Talend ETL process to Microsoft Fabric. In Talend, the same end-to-end process completes in around 3 minutes, whereas in Fabric it is still taking more than 15 minutes earlier, and now around 10 minutes after optimization.
We are continuing to analyze the bottlenecks and would appreciate any additional suggestions or best practices to further improve the performance in Fabric.
- v-kpoloju-msft
Hi udhaya208,
Thank you for the update, and it is great to hear that consolidating the notebooks helped reduce the execution time from 17 minutes to around 10 minutes. This improvement confirms that a significant portion of the earlier delay was related to pipeline orchestration and repeated Spark session initialization overhead.
Since the execution time is still higher compared to Talend, the remaining bottleneck is now likely within the Spark processing layer itself rather than the pipeline orchestration. At this stage, we would recommend reviewing the notebook transformations for expensive operations such as large shuffle operations, repeated joins, repartitioning, unnecessary caching, or multiple actions triggering recomputation. Additionally, please verify whether the workload is experiencing capacity throttling or resource contention in the Fabric workspace. You may also see further improvements by optimizing partitioning strategy, enabling adaptive query execution, reducing small file operations, and reviewing the Spark UI/execution metrics to identify slow stages or skewed tasks.Hope this helps. If you have any questions regarding this, please feel free to reach out us. We will be happy to help.
