Forum Discussion
pipeline slower than notebook
Hi rgsalido,
Thank you for the update.
The behavior you're experiencing is normal when running a notebook through a pipeline. Pipelines typically start a new Spark session for each run, which adds extra time compared to running the notebook manually. Because your pipeline runs every 5 minutes, session startup is likely causing most of the delay.
Even with a session tag applied, Spark may still start a new session if the previous one isn't active or if the compute resources are busy.
To improve performance, you can try these steps:
Use a consistent session tag in the Notebook activity so Fabric can reuse the Spark session when possible.
Enable high-concurrency or session sharing for pipeline notebooks, if your workspace supports it. This helps the pipeline connect to an existing Spark application instead of starting a new one.
Check your Spark pool capacity. If other jobs are using the pool, session startup may be slower because executors aren't available right away.
Review the Spark UI timeline for idle periods, which often show Spark waiting for resources, shuffle, or I/O, rather than issues in your code.
If your pipeline needs to run frequently, consider keeping a warm session active with the same tag so notebook runs can attach to it faster.
Thank you.
I don't have any more notebooks running in that pipeline. The pipeline runs every 5 minutes. Maybe the pipeline+notebook pattern is not the most efficient.
Ideally, the Spark session should be reused for each execution. I have used the tag in the notebook activity, but it doesn't always use the same session.