Forum Discussion

udhaya208's avatar
udhaya208
Regular Visitor
4 months ago
Solved

Fabric Pipeline Performance

Hello Everyone,   We have built a pipeline with a Notebook activity and a Switch activity. Under the Switch activity, we are calling 4 pipelines sequentially, and each pipeline in turn executes a n...
data pipeline
notebook
Spark Job Definition
  • v-kpoloju-msft's avatar
    v-kpoloju-msft
    4 months ago

    Hi udhaya208, 

    Thank you for the update, and it is great to hear that consolidating the notebooks helped reduce the execution time from 17 minutes to around 10 minutes. This improvement confirms that a significant portion of the earlier delay was related to pipeline orchestration and repeated Spark session initialization overhead.

    Since the execution time is still higher compared to Talend, the remaining bottleneck is now likely within the Spark processing layer itself rather than the pipeline orchestration. At this stage, we would recommend reviewing the notebook transformations for expensive operations such as large shuffle operations, repeated joins, repartitioning, unnecessary caching, or multiple actions triggering recomputation. Additionally, please verify whether the workload is experiencing capacity throttling or resource contention in the Fabric workspace. You may also see further improvements by optimizing partitioning strategy, enabling adaptive query execution, reducing small file operations, and reviewing the Spark UI/execution metrics to identify slow stages or skewed tasks.

    Hope this helps. If you have any questions regarding this, please feel free to reach out us. We will be happy to help.

     

v-kpoloju-msft's avatar
v-kpoloju-msft
Icon for Community Support rankCommunity Support
4 months ago

Hi udhaya208,
Thank you for reaching out to the Microsoft Fabric Community Forum.

The notebooks themselves do not appear to be the primary bottleneck, since they complete in less than a minute when executed independently. The increased runtime is most likely caused by pipeline orchestration overhead and Spark session initialization when the notebooks are triggered sequentially through multiple pipelines. In the current design, each notebook execution may be creating a separate Spark session, which can add several minutes of startup latency and significantly increase the total pipeline execution time.

To improve performance, we recommend reducing nested pipeline execution where possible and running notebooks directly within a single pipeline. Additionally, if there are no dependencies between the notebooks, consider running them in parallel instead of sequentially to reduce cumulative delays. Enabling High Concurrency mode can also help by allowing notebook activities to share the same Spark session and minimize repeated startup overhead. Please review the following Microsoft documentation for detailed guidance on optimizing notebook execution and Spark session reuse in Fabric pipelines.

Please refer to the below links:
1. Configure high concurrency mode for notebooks in pipelines - Microsoft Fabric | Microsoft Learn  
2. High concurrency mode in Apache Spark compute for Fabric - Microsoft Fabric | Microsoft Learn
3. Notebook activity - Microsoft Fabric | Microsoft Learn

Hope this helps. If you have any questions regarding this, please feel free to reach out us. We will be happy to help.