notebook queued time

kangwen1 — Thu, 15 Aug 2024 14:20:59 GMT

When I tried to trigger the notebook by the pipeline, I always find that though the notebook execution time is 10 minutes, it's possible the queued duration is 2 minutes.

Why are there this kind of queued duration? How to reduce it?

Re: notebook queued time

frithjof_v — Thu, 15 Aug 2024 18:22:45 GMT

Is the notebook attached to a starter pool or a custom pool?

Re: notebook queued time

Anonymous — Fri, 16 Aug 2024 01:34:07 GMT

Hi @kangwen1 ,

Thanks for the reply from frithjof_v .

There are several possible reasons for queuing.

If the resources required to execute the notebook are occupied, the job will be queued until the resources are free.

There may be a limit to the number of concurrent runs that can be made, resulting in new jobs waiting in the queue.

You can refer to the following documentation for more details on concurrency limits:

Concurrency limits and queueing in Apache Spark for Fabric - Microsoft Fabric | Microsoft Learn

To reduce queuing time, I have a few suggestions here.

Running tasks during periods of low demand can help reduce queuing time, for example, most of the evening is more free than the morning.

You can configure high concurrency mode for notebooks.

This document provides steps on how to configure high concurrency mode for notebooks:

Configure high concurrency mode for notebooks - Microsoft Fabric | Microsoft Learn

If you have any other questions please feel free to contact me.

Best Regards,
Yang
Community Support Team

If there is any post helps, then please consider Accept it as the solution to help the other members find it more quickly.
If I misunderstand your needs or you still have problems on it, please feel free to let us know. Thanks a lot!

topic Re: notebook queued time in Data Engineering

notebook queued time

Re: notebook queued time

Re: notebook queued time