Forum Discussion
Helper INotebook has 98% idle time in pipeline
HI P_work ,
When the same notebook behaves differently inside and outside the pipeline. That’s a classic case of orchestration overhead and environment provisioning latency.
Even though the custom environment seems lightweight, the pipeline engine treats it as a separate provisioning task. That’s why you see the 4-minute delay, it's not just the notebook, it's the orchestration and environment spin-up
Configure your pipeline to use pre-warmed clusters or instance pools. This avoids cold starts and can shave minutes off startup time.
Instead of calling child pipelines in a loop, use APIs like dbutils.notebook.run() (Databricks) or mssparkutils.notebook.run() (Microsoft Fabric). These maintain context and reduce orchestration delays.
Community SupportHI P_work ,
When the same notebook behaves differently inside and outside the pipeline. That’s a classic case of orchestration overhead and environment provisioning latency.
Even though the custom environment seems lightweight, the pipeline engine treats it as a separate provisioning task. That’s why you see the 4-minute delay, it's not just the notebook, it's the orchestration and environment spin-up
Configure your pipeline to use pre-warmed clusters or instance pools. This avoids cold starts and can shave minutes off startup time.
Instead of calling child pipelines in a loop, use APIs like dbutils.notebook.run() (Databricks) or mssparkutils.notebook.run() (Microsoft Fabric). These maintain context and reduce orchestration delays.