Forum Discussion
Notebooks stuck in Starting state in Data Pipelines
Hi, this is usually happening on times where we have multiple notebooks running (big team with regular batch jobs)
This is an example where you can see the Activity has been in progress for 3 minutes:
but clicking on the Notebook activity will open up this:
You see the Notebook is still in the Starting state. If I leave it be, it will stay like this for hours, never executing anything.
If I try to look for the Notebook snapshot, it will show there is no job, the job has failed or the job was stopped, despite the Pipeline still showing that Notebook as In Progress. This does seem to be happening for Notebooks in Workspaces with High Concurrency in Pipelines turned on, but I haven't confirmed if this is exclusive to that or just more common. This defeats the purpose of that feature.
Got a perfect example today. Mind you, we get multiple notebooks with this same scenario.
Stuck In Progress within Pipeline.
Activity Details show it's still Starting
Spark Monitoring shows the Spark Job actually timed out at 20 minutes
This is all for the same Notebook run within a Pipeline.