Forum Discussion
InvalidHttpRequestToLivy: [TooManyRequestsForCapacity]
Can you try to create a custom spark pool for the notebooks and set the node size to small, with autoscale and dynamically allocate executors swtiched off and set the values to 1 or 2? The standard spark pool is already consuming too much for an F2. You will need to connect the custom spark pool to your notebooks and try running it again.
Hello,
Thank you for your reply.
When I look in the metrics app everything seems to be ok. I am not fully aware how to read the app but all graphs tops at 50%. I have no graphs going above 100% utilization. I have no data on overages.
The link you send me seems to be specific for data warehouses. I do have a data warehouse in my workspace but there is nothing in it. I only run pyspark notebooks on a lakehouse and I only run one notebook at a time. I have no jobs scheduled.
Under these circumstances I find it odd that I should be overutilizing the capacity.
Can you try to create a custom spark pool for the notebooks and set the node size to small, with autoscale and dynamically allocate executors swtiched off and set the values to 1 or 2? The standard spark pool is already consuming too much for an F2. You will need to connect the custom spark pool to your notebooks and try running it again.
Here's the link for creating a custom spark pool: https://learn.microsoft.com/en-us/fabric/data-engineering/create-custom-spark-pools
So, I managed to create a separate spark spool and environment. That seems to have helped. I can now consistently run different notebooks without the "TooManyReqeustsForCapacity" error. So the problem seems to be the starter pool ... and my lack of knowledge on utilizing these spark pools.
Yes, I need to acquire more knowledge. It is still odd to me that the starterpool do not work initially but then somehow frees up capacity and can run for almost the entirety of the day.
Thank you for your help. You saved my day 🙂