Forum Discussion
Inconsistent Execution Times in Microsoft Fabric Pipeline
I think there are two distinct issues at play here, one you have noticed and one you might not be aware of.
Issue #1: Inconsistent processing time between invocations of any particular notebook:
This can be caused by a few things, including the time it takes to create a spark session (though, that shouldn't take more than a few minutes). You could also be hitting your spark pool max, or your Fabric capacity maximums. If you have access, you can view the compute usage in the Capacity Metrics Report. If you open up the individual notebook activites in the monitoring details, you can go to the spark monitoring, and there is a "resources (preview)" tab in there that should show you how efficiently your notebook utilized its allocated resources over the duration of its execution. Here you would also see if your spark pool was running out of resources, so notebooks were 'waiting' or 'queueing' for the pools resources to free up so they could allocate them.
There is also the real possibility that some of your data ingestions include more data than others. Which logically, you might expect would take longer to handle. Have you looked into the amount of data being written/read by these notebooks that are taking longer?
There is also the possibility that the API doesn't like you hitting it with a bunch of request very quickly, and might be delaying their responses. Have you logged the API calls from the notebook? You might want to look at when the API was called, when spark got a response, etc.
Issue #2: Notebook parallelization via the Pipeline for-each:
This is what I think you should look into if you really want to increase performance. The way you are executing one notebook at a time, in its own session, is not an effective use of the notebook's ability to perform dynamic parallelization and resource allocation on its own. I highly suggest using one invocation of a notebook which calls usesmssparkutils.notebook.runMultiple() to execute multiple invocations of a notebook, all in a single spark session, rather than simply calling one notebooks repeatedly in a for-each loop. This way, spark is able to really leverage its ability to perform concurrent tasks efficiently. This is a pretty decent article on the subject, which includes some examples of this method (and others) and has good explanations:
https://learn-it-all.medium.com/parallelism-in-spark-notebook-execution-in-microsoft-fabric-8fb6ac3f79be
Best of luck!
I think there are two distinct issues at play here, one you have noticed and one you might not be aware of.
Issue #1: Inconsistent processing time between invocations of any particular notebook:
This can be caused by a few things, including the time it takes to create a spark session (though, that shouldn't take more than a few minutes). You could also be hitting your spark pool max, or your Fabric capacity maximums. If you have access, you can view the compute usage in the Capacity Metrics Report. If you open up the individual notebook activites in the monitoring details, you can go to the spark monitoring, and there is a "resources (preview)" tab in there that should show you how efficiently your notebook utilized its allocated resources over the duration of its execution. Here you would also see if your spark pool was running out of resources, so notebooks were 'waiting' or 'queueing' for the pools resources to free up so they could allocate them.
There is also the real possibility that some of your data ingestions include more data than others. Which logically, you might expect would take longer to handle. Have you looked into the amount of data being written/read by these notebooks that are taking longer?
There is also the possibility that the API doesn't like you hitting it with a bunch of request very quickly, and might be delaying their responses. Have you logged the API calls from the notebook? You might want to look at when the API was called, when spark got a response, etc.
Issue #2: Notebook parallelization via the Pipeline for-each:
This is what I think you should look into if you really want to increase performance. The way you are executing one notebook at a time, in its own session, is not an effective use of the notebook's ability to perform dynamic parallelization and resource allocation on its own. I highly suggest using one invocation of a notebook which calls usesmssparkutils.notebook.runMultiple() to execute multiple invocations of a notebook, all in a single spark session, rather than simply calling one notebooks repeatedly in a for-each loop. This way, spark is able to really leverage its ability to perform concurrent tasks efficiently. This is a pretty decent article on the subject, which includes some examples of this method (and others) and has good explanations:
https://learn-it-all.medium.com/parallelism-in-spark-notebook-execution-in-microsoft-fabric-8fb6ac3f79be
Best of luck!
Thank you so much for your insightful response! Your detailed explanation of the two issues is incredibly helpful. I'll definitely look into the compute usage and check the Capacity Metrics Report to see if we're hitting any resource limits. The tip about the "resources (preview)" tab in the spark monitoring is great—I wasn't aware of that feature.
Regarding the data ingestion sizes and API call logs, those are excellent points that I hadn't fully considered. I'll investigate the data volumes and monitor the API interactions to see if they're contributing to the inconsistent processing times.
Your suggestion on notebook parallelization using usesmssparkutils.notebook.runMultiple() is exactly what I needed. Consolidating the tasks into a single Spark session sounds more efficient. I'll read the article you shared for more insight.
Thanks again for your assistance and the best wishes!