Forum Discussion
Helper INotebook has 98% idle time in pipeline
HI P_work ,
When the same notebook behaves differently inside and outside the pipeline. That’s a classic case of orchestration overhead and environment provisioning latency.
Even though the custom environment seems lightweight, the pipeline engine treats it as a separate provisioning task. That’s why you see the 4-minute delay, it's not just the notebook, it's the orchestration and environment spin-up
Configure your pipeline to use pre-warmed clusters or instance pools. This avoids cold starts and can shave minutes off startup time.
Instead of calling child pipelines in a loop, use APIs like dbutils.notebook.run() (Databricks) or mssparkutils.notebook.run() (Microsoft Fabric). These maintain context and reduce orchestration delays.
Community SupportHi P_work ,
We just wanted to follow up to ensure that your issue has been fully resolved. If there are any outstanding questions or if you need further assistance with anything, please don’t hesitate to reach out. We’re always here and happy to help in any way we can.
Your satisfaction is important to us, and we want to make sure you have everything you need.
Warm regards,
Chaithra E.
P_workI believe my issue is not idle time, but the startup time of custom environments. Default environment startup is roughly 20 secs, custom environment (appears the same as the default with one public library loaded/published) 2 minutes notebook standalone, 4 minutes notebook in pipeline.