Forum Discussion
Helper IInvalidHttpRequestToLivy: [TooManyRequestsForCapacity]
Can you try to create a custom spark pool for the notebooks and set the node size to small, with autoscale and dynamically allocate executors swtiched off and set the values to 1 or 2? The standard spark pool is already consuming too much for an F2. You will need to connect the custom spark pool to your notebooks and try running it again.
HI MartinFM,
I'd like to suggest you take a look at the following links about the spark job and semantic model limitations of different SKU:
Concurrency limits and queueing in Apache Spark for Microsoft Fabric
What is Power BI Premium? - Power BI | Microsoft Learn
According to the document, F2 SKU queue limit is 4. You may need to upgrade the capacity SKU allow more requests.
| Fabric capacity SKU | Equivalent Power BI SKU | Spark VCores | Max Spark VCores with Burst Factor | Queue limit |
|---|---|---|---|---|
| F2 | - | 4 | 20 | 4 |
| F4 | - | 8 | 24 | 4 |
| F8 | - | 16 | 48 | 8 |
| F16 | - | 32 | 96 | 16 |
| F32 | - | 64 | 192 | 32 |
| F64 | P1 | 128 | 384 | 64 |
| F128 | P2 | 256 | 768 | 128 |
| F256 | P3 | 512 | 1536 | 256 |
| F512 | P4 | 1024 | 3072 | 512 |
| F1024 | - | 2048 | 6144 | 1024 |
| F2048 | - | 4096 | 12288 | 2048 |
| Trial Capacity | P1 | 128 | 128 | NA |
Regards,
Xiaoxin Sheng
Helper IHello,
Thank you for your reply.
So, to be more precise, this is a very simple Fabric instance. There are no semantic models. Update of the default semantic model has been switched off. I am only running on the default starter pool and no parameters have been changed for that. There are no scheduled jobs. There is a single lakehouse with maybe 100 quite small files in it. There is 6 runnable notebooks running Pyspark, all manually started. I only run 1 notebook at a time.
I have installed the Fabric Metrics App. All graphs tops at 50% utilization. I have no data on overages.
Given these circumstances I find it odd that I should be overutilizing the capacity. I would have expected to still have an abundance of compute.
I can confirm that the problem persists.