topic Re: [TooManyRequestsForCapacity] This spark job can't be run because you have hit a spark compute or API in Data Engineering

[TooManyRequestsForCapacity] This spark job can't be run because you have hit a spark compute or API

SivaReddy2468 — Fri, 06 Sep 2024 07:17:06 GMT

Hello,
I'm getting the below error when I try to run the two notebooks parallelly.
This is the first notebook running with Standard Session

When I try to run the other notebook, i'm getting the error

I tried using the High Concurrent Seesion. But, Still seeing the same error
Error:
"[TooManyRequestsForCapacity] This spark job can't be run because you have hit a spark compute or API rate limit. To run this spark job, cancel an active Spark job through the Monitoring hub, choose a larger capacity SKU, or try again later. HTTP status code: 430 {Learn more} HTTP status code: 430."

Note: I'm using the Free trial account

Re: [TooManyRequestsForCapacity] This spark job can't be run because you have hit a spark compute or API

FabianSchut — Fri, 06 Sep 2024 08:42:37 GMT

You could try to decrease the Spark Compute for the entire workspace in these settings:
https://learn.microsoft.com/en-us/fabric/data-engineering/create-custom-spark-pools.

When you switch to a small node size and disable autoscale and set a fixed number of executors, you could try to find the best configuration to run your two notebooks while still running your code within reasonable time.

Instead of doing this for the whole workspace, you could also create a custom environment and set your notebooks to that environment. You find the documentation for the custom environment here: https://learn.microsoft.com/en-us/fabric/data-engineering/create-and-use-environment

Re: [TooManyRequestsForCapacity] This spark job can't be run because you have hit a spark compute or API

Anonymous — Thu, 19 Sep 2024 09:48:13 GMT

Hi SivaReddy2468

Do you still have this error? Was adjusting the Spark Compute settings in Workspace helpful?

Best Regards,
Jing