Forum Discussion

Anonymous's avatar
Anonymous
Not applicable
2 years ago
Solved

Configuring queue in Fabric

Hi all!

Does anyone know how the queue works for spark jobs in Fabric?

 

I have a F2 capacity, with 4 Spark VCores and at maximum 20 VCores with bursting factor. According to the documentation, I have a queue limit of 4. https://learn.microsoft.com/en-us/fabric/data-engineering/spark-job-concurrency-and-queueing#concurrency-throttling-and-queueing

 

The first question is whether the queue limit is the amount of jobs or the equivalent number of Spark VCores of a job.

 

I built a pipeline to test how many jobs fabric can deal with. With one job using 4 VCores and 20 available VCores at max (bursting factor) it makes sense that 5 jobs run successfully.

 

It seems like that the queue has no effect here as the 6th and 7th job fail with code 430 (TooManyRequestsForCapacity). If the queue limit was the amount of jobs, both should have succeeded and if it was the number of Spark VCores one should have succeeded. Does anyone have an explanation why both jobs fail?

The aim would be that if no capacity is available, a job is executed when the next capacity is free.

 

The documentation also says that jobs are added to the queue when queueing is enabled. So far I have not found the setting to enable queueing, so I expected it being enabled by default.

 

Thanks

Georg

  • Anonymous's avatar
    Anonymous
    2 years ago

    Hi Anonymous ,

    Response from Internal Team -

    " The queuing is not enabled yet in the pipeline. When enabled, the jobs will queue instead of failing (current behavior). The support for queueing notebook jobs is still in development and once enabled - notebook jobs triggered from pipelines and schedulers will be queued and automatically retried when cores become available. Currently we don't have any ETA, but we can expect this feature in upcoming releases. "

    Hope this is helpful. Please let me know incase of further queries.

4 Replies

  • Anonymous's avatar
    Anonymous
    Not applicable

    Hello Anonymous ,

    Thanks for using Fabric Community.
    At this time, we are reaching out to the internal team to get some help on this .
    We will update you once we hear back from them.

    • Anonymous's avatar
      Anonymous
      Not applicable

      Hi Anonymous ,

      Response from Internal Team -

      " The queuing is not enabled yet in the pipeline. When enabled, the jobs will queue instead of failing (current behavior). The support for queueing notebook jobs is still in development and once enabled - notebook jobs triggered from pipelines and schedulers will be queued and automatically retried when cores become available. Currently we don't have any ETA, but we can expect this feature in upcoming releases. "

      Hope this is helpful. Please let me know incase of further queries.

      • Anonymous's avatar
        Anonymous
        Not applicable

        Thanks Anonymous,

        looking forward to see the feature in Fabric soon!

         

        Greetings,

        Georg