Slow starter pool session initialization

[FilipO]

I am currently experimenting with a F2 capacity. Whenever I try to run a notebook it takes somewhere between 2 and 2.5 minutes to start the spark cluster even though I'm using the starter pool. All settings related to Spark are left as default for the workspace. Tried both running things manually as well as triggering the notebook from data factory.

If I go into a trial workspace and run a notebook the spark session is started in just a few seconds. This makes me think that it might be a limitation depending on which capacity you are using? But the documentation gives no indication of this: https://learn.microsoft.com/en-us/fabric/data-engineering/spark-compute

[ramonsuarez]

I spoke with support yesterday and they claimed that 3 to 4 minutes to start running a notebook is normal... And then when I was demonstrating the start of a notebook it simply failed... 😀 

[scca]

Hey there. I'm still having this issue with slow start ups. I notice that this problem doesn't occur when starting a notebook with the starter pool (which fires up within seconds). So if the starter pool can start up that quickly, it doesn't make sense for a custom pool to take much longer, yeah? Imagine the same notebook being used in parallel and each of them taking that many minutes and times that by the number of jobs in the queue waiting to go through the notebook start up wait time... 

[alozovoy]

Yeah, it is furstrating that they can take several minutes to start up.

One option is to use runMultiple to execute your notebooks. You can define the dependencies and then execute all of the notebooks on a single compute. It only has to start up once rather than for each notebook.

Using runMultiple To Orchastrate Notebook Execution in Microsoft Fabric