Run notebooks in parallel

Hi priyankabis ,

Running notebooks in parallel can sometimes lead to longer execution times due to a variety of factors such as resource contention, overhead of managing parallel tasks, or inefficient configurations.

For your first question, I think first you need to make sure that the cluster has enough resources (CPU, memory) to handle multiple notebooks running in parallel. If resources are limited, tasks may compete for resources and cause delays. Secondly you need to check if the cluster configuration can handle parallel execution efficiently. Running notebooks through a pipeline may incur additional overhead compared to running them individually. This could be due to setup and teardown times, data transfers between stages, or other pipeline management tasks.

For your second question, I think you can Implement a clustering policy that forces all notebooks to use a shared cluster. This helps prevent the creation of multiple clusters and ensures efficient utilization of resources. If laptops are queuing, it may be due to the capacity limitations of the cluster. Increasing the cluster size or optimizing the laptops to run more efficiently can help reduce queuing.

Best Regards

Yilong Zhou

If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.