Forum Discussion
Notebooks taking a considerable time to run when deployed in data pipelines
- 2 years ago
I think using custom environments increase the start-up time.
Also, if you're not using the starter pools, it will increase the start-up time.
I'm not sure about the queuing you're experiencing. I'm not sure if that is due to using a custom environment or if it's about something else.
Is this the only notebook in your pipeline? Or do you run another notebook just before this one?
Could it be that you or another user is running a notebook or somehow using spark at the same time?
Here is a couple of articles about queuing:
https://learn.microsoft.com/en-us/fabric/data-engineering/job-queueing-for-fabric-spark
https://learn.microsoft.com/en-us/fabric/data-engineering/spark-job-concurrency-and-queueing
I think using custom environments increase the start-up time.
Also, if you're not using the starter pools, it will increase the start-up time.
I'm not sure about the queuing you're experiencing. I'm not sure if that is due to using a custom environment or if it's about something else.
Is this the only notebook in your pipeline? Or do you run another notebook just before this one?
Could it be that you or another user is running a notebook or somehow using spark at the same time?
Here is a couple of articles about queuing:
https://learn.microsoft.com/en-us/fabric/data-engineering/job-queueing-for-fabric-spark
https://learn.microsoft.com/en-us/fabric/data-engineering/spark-job-concurrency-and-queueing