Notebooks taking a considerable time to run when deployed in data pipelines

[choran]

Hi all!

I am currently experiencing substantial runtimes for my data pipelines due to long runtimes for my notebooks. 

A breakdown of the time that it takes for a notebook to run (when no other pipelines, or notebooks are running, that may consume Spark capacity):

- Notebook activity overall runtime: 4-7mins

- Actual time to run activities within notebook: Less than 60s

I have shown an example screenshot when I view the run details of the notebook activity from within the pipeline: 

- The notebook is being run in a custom environment as I need to import the AzureOpenAI library. 

- I am currently utilising F8 SKU.

I think it is taking a long-time to run as it takes time to connect to a Spark cluster. Is there a way of speeding this up with configurations, or is it a case of increasing the SKU?

[frithjof_v · Accepted solution]

I think using custom environments increase the start-up time.

Also, if you're not using the starter pools, it will increase the start-up time.

I'm not sure about the queuing you're experiencing. I'm not sure if that is due to using a custom environment or if it's about something else. 

Is this the only notebook in your pipeline? Or do you run another notebook just before this one?

Could it be that you or another user is running a notebook or somehow using spark at the same time?

Here is a couple of articles about queuing:

https://learn.microsoft.com/en-us/fabric/data-engineering/job-queueing-for-fabric-spark

https://learn.microsoft.com/en-us/fabric/data-engineering/spark-job-concurrency-and-queueing