Forum Discussion
Fabric Pipeline Performance
Hi udhaya208,
Thank you for the update, and it is great to hear that consolidating the notebooks helped reduce the execution time from 17 minutes to around 10 minutes. This improvement confirms that a significant portion of the earlier delay was related to pipeline orchestration and repeated Spark session initialization overhead.
Since the execution time is still higher compared to Talend, the remaining bottleneck is now likely within the Spark processing layer itself rather than the pipeline orchestration. At this stage, we would recommend reviewing the notebook transformations for expensive operations such as large shuffle operations, repeated joins, repartitioning, unnecessary caching, or multiple actions triggering recomputation. Additionally, please verify whether the workload is experiencing capacity throttling or resource contention in the Fabric workspace. You may also see further improvements by optimizing partitioning strategy, enabling adaptive query execution, reducing small file operations, and reviewing the Spark UI/execution metrics to identify slow stages or skewed tasks.Hope this helps. If you have any questions regarding this, please feel free to reach out us. We will be happy to help.
Community SupportHi udhaya208,
Just checking in to see if the issue has been resolved on your end. If the earlier suggestions helped, that’s great to hear! And if you’re still facing challenges, feel free to share more details happy to assist further.
Thank you.
Hi v-kpoloju-msft,
Thanks for your suggestion. I really appreciate your Efforts. As recommended, I removed the sequential pipeline execution and consolidated the four notebooks into a single notebook. This optimization reduced the execution time from 17 minutes to 10 minutes for the same volume of records. However, our expectation is to bring it down to less than 5 minutes.
For some background, we are currently migrating the Talend ETL process to Microsoft Fabric. In Talend, the same end-to-end process completes in around 3 minutes, whereas in Fabric it is still taking more than 15 minutes earlier, and now around 10 minutes after optimization.
We are continuing to analyze the bottlenecks and would appreciate any additional suggestions or best practices to further improve the performance in Fabric.
- v-kpoloju-msft
Hi udhaya208,
Thank you for the update, and it is great to hear that consolidating the notebooks helped reduce the execution time from 17 minutes to around 10 minutes. This improvement confirms that a significant portion of the earlier delay was related to pipeline orchestration and repeated Spark session initialization overhead.
Since the execution time is still higher compared to Talend, the remaining bottleneck is now likely within the Spark processing layer itself rather than the pipeline orchestration. At this stage, we would recommend reviewing the notebook transformations for expensive operations such as large shuffle operations, repeated joins, repartitioning, unnecessary caching, or multiple actions triggering recomputation. Additionally, please verify whether the workload is experiencing capacity throttling or resource contention in the Fabric workspace. You may also see further improvements by optimizing partitioning strategy, enabling adaptive query execution, reducing small file operations, and reviewing the Spark UI/execution metrics to identify slow stages or skewed tasks.Hope this helps. If you have any questions regarding this, please feel free to reach out us. We will be happy to help.
- v-kpoloju-msft
Hi udhaya208,
Just wanted to follow up. If the shared guidance worked for you, that’s wonderful hopefully it also helps others looking for similar answers. If there’s anything else you'd like to explore or clarify, don’t hesitate to reach out.Thank you.
- v-kpoloju-msft
Hi udhaya208,
Just checking in to see if the issue has been resolved on your end. If the earlier suggestions helped, that’s great to hear! And if you’re still facing challenges, feel free to share more details happy to assist further.Thank you.