Notebooks stuck in Starting state in Data Pipelines

[JosueMolina]

Has anybody been experiencing an issue where Notebooks get stuck in Starting state when looking at a Data Pipeline Run, but once you enter the Spark Application for that Notebook run, it shows as either Stopped at 20 minutes or failed at 2 or 3 minutes. 

I believe the Stopped or Failed status comes from too many concurrent notebooks (I was under the impression the notebooks would be Queued but that's a separate issue). But my main problem is that the Pipeline is still showing those Notebook activities as In Progress (with the Cluster showing as Starting), so any Notebook activities that follow will never run. I can solve this by setting a Time Out for those activities, but I assume this is a Pipeline bug since this behavior was not common until recently. 

[JhonatanD]

I managed to solve mine to not be stuck, deactivating this option

[JosueMolina]

Right, but I actually need High Concurrency turned on as we depend on this to run a set of small notebooks on parallel. 

[Anonymous]

Hi @JosueMolina ,

Can you please share some more detail information about this issue? They should help us clarify your scenario and test to troubleshoot.

Regards,

Xiaoxin Sheng

[JosueMolina]

Hi, this is usually happening on times where we have multiple notebooks running (big team with regular batch jobs)

This is an example where you can see the Activity has been in progress for 3 minutes:

but clicking on the Notebook activity will open up this:
 

You see the Notebook is still in the Starting state. If I leave it be, it will stay like this for hours, never executing anything.

If I try to look for the Notebook snapshot, it will show there is no job, the job has failed or the job was stopped, despite the Pipeline still showing that Notebook as In Progress. This does seem to be happening for Notebooks in Workspaces with High Concurrency in Pipelines turned on, but I haven't confirmed if this is exclusive to that or just more common. This defeats the purpose of that feature.

[JosueMolina]

Got a perfect example today. Mind you, we get multiple notebooks with this same scenario. 

Stuck In Progress within Pipeline.

Activity Details show it's still Starting 

Spark Monitoring shows the Spark Job actually timed out at 20 minutes

This is all for the same Notebook run within a Pipeline.

[JosueMolina]

What sort of detail?

As mentioned, a notebook activity will show as In Progress inside a Data Pipeline but viewing the specific Spark application will show it is either failed or stopped. I believe this happens because there are too many notebook sessions running at once, though I believe extra Notebooks are meant to be queued and not stuck or failed.

I am running this on an F16 capacity as part of our data pipeline.

[JhonatanD]

My workspaces look like this, too