topic This spark job can't be run because you have hit a spark compute or API rate limit in Data Engineering

This spark job can't be run because you have hit a spark compute or API rate limit

DebbieE — Tue, 02 Jul 2024 10:12:20 GMT

We are still just testing Fabric and because the trial is over, have moved to the most basic fabric capacity.

Im the only one using it.

I run through a notebook and at the end I have a bit of code

# Stop the Spark session

spark.stop()

Which stops the job and it should be the only job running (standard session)

I then go across to the next notebook and get

InvalidHttpRequestToLivy: [TooManyRequestsForCapacity] This spark job can't be run because you have hit a spark compute or API rate limit. To run this spark job, cancel an active Spark job through the Monitoring hub, choose a larger capacity SKU, or try again later. HTTP status code: 430 {Learn more} HTTP status code: 430.

so even though i have cancelled the original job. I can't run the next notebook. If I click on Monitor icon i can see its still in Progress. So I stop it here and try again and it works.

Why didnt my code stop the job. it looked like it did in the notebook? I shouldnt have to manually exit from it surely?

Re: This spark job can't be run because you have hit a spark compute or API rate limit

Anonymous — Wed, 31 Jul 2024 08:44:11 GMT

HI DebbieE,

As the error message mentions, the spark job can't run because this operation hit the computer or API rate limit.

What type of capacity are you use to host these processes? How did you configurate the spark environment and pool settings? Please share some more detail information to help us clarify your scenario and test to troubleshoot.

Reference links:

Configure and manage starter pools in Fabric Spark. - Microsoft Fabric | Microsoft Learn

Compute management in Fabric environments - Microsoft Fabric | Microsoft Learn

Regards,

Xiaoxin Sheng

Re: This spark job can't be run because you have hit a spark compute or API rate limit

Anonymous — Fri, 09 Aug 2024 07:02:29 GMT

Hi DebbieE,

Any update on this? For the job queue limitation you can refer to following document:

Job queueing in Apache Spark for Fabric - Microsoft Fabric | Microsoft Learn

If these processing meet to the limitations, you can consider reducing the total queue amount or upgrade the capacity tier.

Regards,

Xiaoxin Sheng

Re: This spark job can't be run because you have hit a spark compute or API rate limit

DebbieE — Fri, 09 Aug 2024 08:09:19 GMT

As above. Its the lowest capacity.

However. I have

# Stop the Spark session

spark.stop()

At the end of the notebook. And this should stop the spark session. So then I can go on and run the next notebook. It still isnt working. I have to go actoss to Monitor to stop it.

Thats the issue. Why isnt the Spark.stop() code working?

Re: This spark job can't be run because you have hit a spark compute or API rate limit

frithjof_v — Mon, 12 Aug 2024 19:45:37 GMT

My impression is that using spark.stop() doesn't work in Fabric notebooks.

You could try to use mssparkutils.session.stop() instead

https://learn.microsoft.com/en-us/azure/synapse-analytics/spark/microsoft-spark-utilities?pivots=programming-language-python#stop-an-interactive-session

Even if this documentation is for Azure Synapse Analytics, it seems to work, at least for the moment.

Here is the Notebookutils for Fabric, it doesn't mention this function.

https://learn.microsoft.com/en-us/fabric/data-engineering/notebook-utilities

You can also click the Stop button in the Notebook user interface.

If you want to programmatically run the next notebook after the first notebook has finished, you can use the notebookutils.notebook.run() function in Notebookutils.

https://learn.microsoft.com/en-us/fabric/data-engineering/notebook-utilities#reference-a-notebook

Re: This spark job can't be run because you have hit a spark compute or API rate limit

Anonymous — Thu, 15 Aug 2024 08:20:39 GMT

Hi DebbieE,

You can also refer to the following link to trace the notebook usages:

Notebook contextual monitoring and debugging - Microsoft Fabric | Microsoft Learn

Regards,

Xiaoxin Sheng

Re: This spark job can't be run because you have hit a spark compute or API rate limit

MartinFM — Fri, 01 Nov 2024 12:32:59 GMT

I get this error all the time. Nothing is running in Monitor but I still get the error. I have no idea what I have to stop so that I can be allowed to run my notebooks.

I have the lowest capacity F2. We are also evaluating whether or not Fabric could be our future data platform.

Re: This spark job can't be run because you have hit a spark compute or API rate limit

infomod — Thu, 20 Mar 2025 01:05:52 GMT

Any update on this, how to stop a Spark session in MS Fabric notebook?

Re: This spark job can't be run because you have hit a spark compute or API rate limit

infomod — Thu, 20 Mar 2025 01:24:55 GMT

I see this

notebookutils.session.stop()

in the NotebookUtils (former MSSparkUtils) for Fabric - Microsoft Fabric | Microsoft Learn

Seems to work.