Forum Discussion
Helper INotebook has 98% idle time in pipeline
HI P_work ,
When the same notebook behaves differently inside and outside the pipeline. That’s a classic case of orchestration overhead and environment provisioning latency.
Even though the custom environment seems lightweight, the pipeline engine treats it as a separate provisioning task. That’s why you see the 4-minute delay, it's not just the notebook, it's the orchestration and environment spin-up
Configure your pipeline to use pre-warmed clusters or instance pools. This avoids cold starts and can shave minutes off startup time.
Instead of calling child pipelines in a loop, use APIs like dbutils.notebook.run() (Databricks) or mssparkutils.notebook.run() (Microsoft Fabric). These maintain context and reduce orchestration delays.
Super UserHi P_work ,
Is your question on reducing start time of spark session for each notebooks(parent and child) or to reduce idle time?
1. You could leverage High concurrency session , so that your child notebook can utilise the same spark session as parent and it can avoid separate start time for child notebooks
2. You can stop the session by writing code once your notebook job has completed run , it helps in less consumption of CUs. Also you can mention timeout for session in spark settings as default is 20 mins. Wherther your notebook is running for the entire 20 mins or not it will consume CUs. So it is good to mention in spark settings or adding a code to manually stop once the notebook execution is over.
Regards,
Srisakthi