Configuring queue in Fabric

[Anonymous]

Hi all!

Does anyone know how the queue works for spark jobs in Fabric?

I have a F2 capacity, with 4 Spark VCores and at maximum 20 VCores with bursting factor. According to the documentation, I have a queue limit of 4. https://learn.microsoft.com/en-us/fabric/data-engineering/spark-job-concurrency-and-queueing#concurr...

The first question is whether the queue limit is the amount of jobs or the equivalent number of Spark VCores of a job.

I built a pipeline to test how many jobs fabric can deal with. With one job using 4 VCores and 20 available VCores at max (bursting factor) it makes sense that 5 jobs run successfully.

It seems like that the queue has no effect here as the 6th and 7th job fail with code 430 (TooManyRequestsForCapacity). If the queue limit was the amount of jobs, both should have succeeded and if it was the number of Spark VCores one should have succeeded. Does anyone have an explanation why both jobs fail?

The aim would be that if no capacity is available, a job is executed when the next capacity is free.

The documentation also says that jobs are added to the queue when queueing is enabled. So far I have not found the setting to enable queueing, so I expected it being enabled by default.

Thanks

Georg

[Anonymous]

Hello @Anonymous ,

Thanks for using Fabric Community.
At this time, we are reaching out to the internal team to get some help on this .
We will update you once we hear back from them.

[Anonymous · Accepted solution]

Hi @Anonymous ,

Response from Internal Team -

" The queuing is not enabled yet in the pipeline. When enabled, the jobs will queue instead of failing (current behavior). The support for queueing notebook jobs is still in development and once enabled - notebook jobs triggered from pipelines and schedulers will be queued and automatically retried when cores become available. Currently we don't have any ETA, but we can expect this feature in upcoming releases. "

Hope this is helpful. Please let me know incase of further queries.

[Anonymous]

Thanks @Anonymous,

looking forward to see the feature in Fabric soon!

Greetings,

Georg

[Anonymous]

Hi @Anonymous ,

Glad to know that your query got resolved. Please continue using Fabric Community for your further queries.