Spark job never ends

[TheFifthJoin]

We have some SQL code in a pyspark notebook creating a materialised lake view. 

We do this a lot for our medallion architecture but for some reason, this script causes the spark session to create these little jobs that never end (see image). This is the only script it does this for. The script itself runs fine in the correct time and produces the correct data. It also looks like it finishes with a green tick on the code cell, but the spark job remains 'in progress'. The data from the MLV is accessible once the code block finishes, so no locks going on that I can see. It also doesn't look like these jobs are impacting our capacity usage that I can see either. All of the parent jobs seem to contain a child thats complete, but the parent stays in progress forever..

Anyone seen this before and know what they could be?

left long enough, the children jobs turn to this, but the timer on the parent keeps running

The only difference between ones we've done so far and this on in particular is that there are 2 CTE's in the SQL.. didnt think those were an issue

If I F5 the screen at any point after the green tick shows up in the code cell, it all goes away. Is this just weird GUI feeding back spark cleanup or somehting I can ignore or do I need to address something here?

Thanks 

[spaceman127]

Hi @TheFifthJoin ,

Interesting,
but does the notebook report “Succeeded” at some point in the pipeline, or does the notebook time out?

As far as I understand, everything is running smoothly.
I would also suspect a GUI bug here. Personally, I haven't noticed this yet.

I'll keep an eye out for it.

Best regards

[TheFifthJoin]

it never returns 'Succeeded' on the Spark Job line. It'll sit there with the parent jobs on a running timer, but the child job under it will eventually time out and show 'unable to load stages' after about 20 minutes (picture 2 in the original post). 

[tayloramy]

Hi @TheFifthJoin, 

Very interesting. This looks like a UI bug to me, especially if once you refresh it all goes away. 

Though it is interesting that you're only seeing this UI bug on one notebook and not on all of them. 

Is there anything special about the configuration of this specific notebook? Is it using a different spark pool or environment by chance? 

If you found this helpful, consider giving some Kudos. If I answered your question or solved your problem, mark this post as the solution.

[TheFifthJoin]

I have an MLV environment set up with some resource scripts, so all of them use that - no change there.

The only difference is the size of the script - getting into Gold so we're looking at bigger scripts to denormalize our structure, and some CTE's..

Everything else is the same (I use a template script for MLV creation so apart from the actual SQL script between the quotations in spark.sql(""), the rest is identical)

[tayloramy · Accepted solution]

yeah this seems to be a UI bug. Open a ticket to report it is my suggestion. 

If you found this helpful, consider giving some Kudos. If I answered your question or solved your problem, mark this post as the solution.

[TheFifthJoin]

Ticket raised with MS - I'll post an updated reply if it turns out to be anything other than a GUI issue, incase anyone else stubles across this same issue.