Forum Discussion
[TooManyRequestsForCapacity] issue with multiple notebook users on an F128 Fabric instance
We fixed the concurrent notebook issue by using the runMultiple method like the example below.
You create a DAG with the order and dependencies when calling the notebooks. They all end up using the same spark compute and environment and the concurrency issue is mitigated in the backend.
mssparkutils.notebook.runMultiple(DAG, {"p1":True, "p2":"spectral", "p3":11})
runMultiple , on the other hand, allows you to create a Direct Acyclic Graph (DAG) of notebooks to execute notebooks in parallel and in specified order, similar to a pipeline run except in a notebook. The advantages here are that you can:
programmatically define the execution order
run notebooks in parallel or in squence
define dependencies between notebook runs
efficient use of the compute resources as it will use the compute of the orchestrating notebook
Depending on the F SKU and the node size, you may get TooManyRequestsForCapacity error if you run multiple notebooks at the same time because of concurrency limits. Since runMultiple uses the same compute, you can mitigate this error. Advancing Analytics has an excellent blog that goes in-depth into this. My colleague Will Crayger also has also researched this topic and summarized his findings here.
Using runMultiple To Orchastrate Notebook Execution in Microsoft Fabric
Thanks for replying. I have seen this blog post. The issue we were having wasn't that we had 1 'user'/process wanting to execute multiple notebooks, but multiple people each wanting to run notebooks in different workspaces. We were running a 'learn Fabric' workshop.
Hello are the 16 notebooks writing to the same lakehouse? or even the same table in a lakehouse?
Did you check the Fabric Capacity Metrics App? What can you see in the Utililzation, Throttling and overages graphs? Check out the following blog to better understand the different metrics being reported.
