Forum Discussion
Helper IInvalidHttpRequestToLivy: [TooManyRequestsForCapacity]
Can you try to create a custom spark pool for the notebooks and set the node size to small, with autoscale and dynamically allocate executors swtiched off and set the values to 1 or 2? The standard spark pool is already consuming too much for an F2. You will need to connect the custom spark pool to your notebooks and try running it again.
Solution SageHi, Fabric applies "smoothing" to jobs that exceed their allocated compute units (CUs), allowing them to "borrow" CUs from future periods. For scheduled and background jobs, this borrowing can extend up to 24 hours: https://learn.microsoft.com/en-us/fabric/data-warehouse/compute-capacity-smoothing-throttling.
This could explain why jobs sometimes run unpredictably, like in the morning. This could explain why it will run at random times in the morning. I recommend checking Fabric's capacity metrics for any peak within the 24 hours leading up to the error.
MartinFMHello,
Thank you for your reply.
When I look in the metrics app everything seems to be ok. I am not fully aware how to read the app but all graphs tops at 50%. I have no graphs going above 100% utilization. I have no data on overages.
The link you send me seems to be specific for data warehouses. I do have a data warehouse in my workspace but there is nothing in it. I only run pyspark notebooks on a lakehouse and I only run one notebook at a time. I have no jobs scheduled.
Under these circumstances I find it odd that I should be overutilizing the capacity.
- FabianSchut
Can you try to create a custom spark pool for the notebooks and set the node size to small, with autoscale and dynamically allocate executors swtiched off and set the values to 1 or 2? The standard spark pool is already consuming too much for an F2. You will need to connect the custom spark pool to your notebooks and try running it again.
- FabianSchut
Here's the link for creating a custom spark pool: https://learn.microsoft.com/en-us/fabric/data-engineering/create-custom-spark-pools