Forum Discussion

udhaya208's avatar
udhaya208
Regular Visitor
4 months ago
Solved

Fabric Pipeline Performance

Hello Everyone,   We have built a pipeline with a Notebook activity and a Switch activity. Under the Switch activity, we are calling 4 pipelines sequentially, and each pipeline in turn executes a n...
data pipeline
notebook
Spark Job Definition
  • v-kpoloju-msft's avatar
    v-kpoloju-msft
    3 months ago

    Hi udhaya208, 

    Thank you for the update, and it is great to hear that consolidating the notebooks helped reduce the execution time from 17 minutes to around 10 minutes. This improvement confirms that a significant portion of the earlier delay was related to pipeline orchestration and repeated Spark session initialization overhead.

    Since the execution time is still higher compared to Talend, the remaining bottleneck is now likely within the Spark processing layer itself rather than the pipeline orchestration. At this stage, we would recommend reviewing the notebook transformations for expensive operations such as large shuffle operations, repeated joins, repartitioning, unnecessary caching, or multiple actions triggering recomputation. Additionally, please verify whether the workload is experiencing capacity throttling or resource contention in the Fabric workspace. You may also see further improvements by optimizing partitioning strategy, enabling adaptive query execution, reducing small file operations, and reviewing the Spark UI/execution metrics to identify slow stages or skewed tasks.

    Hope this helps. If you have any questions regarding this, please feel free to reach out us. We will be happy to help.

     

Areeshabaloch7's avatar
Areeshabaloch7
Frequent Visitor
1 month ago

Hi udhaya208,

 

For better performance with multiple pipelines in Switch activity, try these:

 

1. **Parallel instead of Sequential**: If those 4 pipelines don't depend on each other, run them in parallel using "Execute Pipeline" activities without dependency.

2. **Notebook Tuning**: Check Spark pool size. Use bigger node size if data is large.

3. **Cache Data**: If same data is used in 4 pipelines, cache it in Lakehouse first.

 

This will reduce total runtime.

Let me know your pipeline run time, I can suggest more.