Forum Discussion
Fabric Pipeline Performance
Hello Everyone,
We have built a pipeline with a Notebook activity and a Switch activity. Under the Switch activity, we are calling 4 pipelines sequentially, and each pipeline in turn executes a notebook.
Currently, when we execute the notebooks individually, each notebook completes in less than a minute. However, when the same notebooks are triggered through the pipeline, each execution takes around 4–5 minutes. As a result, the overall pipeline execution time has increased to approximately 15 minutes, whereas our expectation is to complete the entire process in under 5 minutes.
Could you please provide some guidance or best practices to help improve the execution performance and reduce the overall pipeline runtime?
Thanks
Hi udhaya208,
Thank you for the update, and it is great to hear that consolidating the notebooks helped reduce the execution time from 17 minutes to around 10 minutes. This improvement confirms that a significant portion of the earlier delay was related to pipeline orchestration and repeated Spark session initialization overhead.
Since the execution time is still higher compared to Talend, the remaining bottleneck is now likely within the Spark processing layer itself rather than the pipeline orchestration. At this stage, we would recommend reviewing the notebook transformations for expensive operations such as large shuffle operations, repeated joins, repartitioning, unnecessary caching, or multiple actions triggering recomputation. Additionally, please verify whether the workload is experiencing capacity throttling or resource contention in the Fabric workspace. You may also see further improvements by optimizing partitioning strategy, enabling adaptive query execution, reducing small file operations, and reviewing the Spark UI/execution metrics to identify slow stages or skewed tasks.Hope this helps. If you have any questions regarding this, please feel free to reach out us. We will be happy to help.
8 Replies
Hi udhaya208,
Thank you for reaching out to the Microsoft Fabric Community Forum.The notebooks themselves do not appear to be the primary bottleneck, since they complete in less than a minute when executed independently. The increased runtime is most likely caused by pipeline orchestration overhead and Spark session initialization when the notebooks are triggered sequentially through multiple pipelines. In the current design, each notebook execution may be creating a separate Spark session, which can add several minutes of startup latency and significantly increase the total pipeline execution time.
To improve performance, we recommend reducing nested pipeline execution where possible and running notebooks directly within a single pipeline. Additionally, if there are no dependencies between the notebooks, consider running them in parallel instead of sequentially to reduce cumulative delays. Enabling High Concurrency mode can also help by allowing notebook activities to share the same Spark session and minimize repeated startup overhead. Please review the following Microsoft documentation for detailed guidance on optimizing notebook execution and Spark session reuse in Fabric pipelines.Please refer to the below links:
1. Configure high concurrency mode for notebooks in pipelines - Microsoft Fabric | Microsoft Learn
2. High concurrency mode in Apache Spark compute for Fabric - Microsoft Fabric | Microsoft Learn
3. Notebook activity - Microsoft Fabric | Microsoft LearnHope this helps. If you have any questions regarding this, please feel free to reach out us. We will be happy to help.
Hi udhaya208,
Just checking in to see if the issue has been resolved on your end. If the earlier suggestions helped, that’s great to hear! And if you’re still facing challenges, feel free to share more details happy to assist further.Thank you.
Hi v-kpoloju-msft,
Thanks for your suggestion. I really appreciate your Efforts. As recommended, I removed the sequential pipeline execution and consolidated the four notebooks into a single notebook. This optimization reduced the execution time from 17 minutes to 10 minutes for the same volume of records. However, our expectation is to bring it down to less than 5 minutes.
For some background, we are currently migrating the Talend ETL process to Microsoft Fabric. In Talend, the same end-to-end process completes in around 3 minutes, whereas in Fabric it is still taking more than 15 minutes earlier, and now around 10 minutes after optimization.
We are continuing to analyze the bottlenecks and would appreciate any additional suggestions or best practices to further improve the performance in Fabric.
Hi udhaya208,
Configuring high concurrency spark sessions is going to help as long as your notebooks all have the same default lakehouse.
If they have different default lakehouses, then high concurrency spark sessions won't help. If this is the case, I'd recommend running the notebooks in parallel instead of concurrently.
Hi udhaya208,
For better performance with multiple pipelines in Switch activity, try these:
1. **Parallel instead of Sequential**: If those 4 pipelines don't depend on each other, run them in parallel using "Execute Pipeline" activities without dependency.
2. **Notebook Tuning**: Check Spark pool size. Use bigger node size if data is large.
3. **Cache Data**: If same data is used in 4 pipelines, cache it in Lakehouse first.
This will reduce total runtime.
Let me know your pipeline run time, I can suggest more.
