Forum Discussion
Helper IInvalidHttpRequestToLivy: [TooManyRequestsForCapacity]
Can you try to create a custom spark pool for the notebooks and set the node size to small, with autoscale and dynamically allocate executors swtiched off and set the values to 1 or 2? The standard spark pool is already consuming too much for an F2. You will need to connect the custom spark pool to your notebooks and try running it again.
Solution SageHi, Fabric applies "smoothing" to jobs that exceed their allocated compute units (CUs), allowing them to "borrow" CUs from future periods. For scheduled and background jobs, this borrowing can extend up to 24 hours: https://learn.microsoft.com/en-us/fabric/data-warehouse/compute-capacity-smoothing-throttling.
This could explain why jobs sometimes run unpredictably, like in the morning. This could explain why it will run at random times in the morning. I recommend checking Fabric's capacity metrics for any peak within the 24 hours leading up to the error.
Helper IHello,
Thank you for your reply.
When I look in the metrics app everything seems to be ok. I am not fully aware how to read the app but all graphs tops at 50%. I have no graphs going above 100% utilization. I have no data on overages.
The link you send me seems to be specific for data warehouses. I do have a data warehouse in my workspace but there is nothing in it. I only run pyspark notebooks on a lakehouse and I only run one notebook at a time. I have no jobs scheduled.
Under these circumstances I find it odd that I should be overutilizing the capacity.
- FabianSchut
Can you try to create a custom spark pool for the notebooks and set the node size to small, with autoscale and dynamically allocate executors swtiched off and set the values to 1 or 2? The standard spark pool is already consuming too much for an F2. You will need to connect the custom spark pool to your notebooks and try running it again.
- FabianSchut
Here's the link for creating a custom spark pool: https://learn.microsoft.com/en-us/fabric/data-engineering/create-custom-spark-pools
- MartinFM
So, I managed to create a separate spark spool and environment. That seems to have helped. I can now consistently run different notebooks without the "TooManyReqeustsForCapacity" error. So the problem seems to be the starter pool ... and my lack of knowledge on utilizing these spark pools.
Yes, I need to acquire more knowledge. It is still odd to me that the starterpool do not work initially but then somehow frees up capacity and can run for almost the entirety of the day.
Thank you for your help. You saved my day 🙂