Forum Discussion
Helper INotebook has 98% idle time in pipeline
HI P_work ,
When the same notebook behaves differently inside and outside the pipeline. That’s a classic case of orchestration overhead and environment provisioning latency.
Even though the custom environment seems lightweight, the pipeline engine treats it as a separate provisioning task. That’s why you see the 4-minute delay, it's not just the notebook, it's the orchestration and environment spin-up
Configure your pipeline to use pre-warmed clusters or instance pools. This avoids cold starts and can shave minutes off startup time.
Instead of calling child pipelines in a loop, use APIs like dbutils.notebook.run() (Databricks) or mssparkutils.notebook.run() (Microsoft Fabric). These maintain context and reduce orchestration delays.
Community SupportHi P_work ,
Once your notebook has completed its run, explicitly stopping the Spark session (using spark.stop()) ensures resources are freed promptly. You can set a shorter timeout for idle Spark sessions via the Spark config [spark.databricks.cluster.profile.serverless.idleTimeout or equivalent], instead of relying on the default (often 20 minutes).
When setting up your pipeline, enable cluster reuse across pipeline tasks to avoid launching a fresh cluster for each notebook or job. alternatively, use pre-warmed instance pools to reduce cluster start times especially in job cluster mode.
Consider executing child notebooks directly within a single pipeline, using a notebook task group or API like mssparkutils.notebook.run() (on Azure Synapse/Microsoft Fabric) or dbutils.notebook.run() (on Databricks). This maintains context, avoids orchestration overhead, and allows better Spark session sharing.
I hope this helps.
Best Regards,
Chaithra E.