Forum Discussion
Spark job never ends
We have some SQL code in a pyspark notebook creating a materialised lake view.
We do this a lot for our medallion architecture but for some reason, this script causes the spark session to create these little jobs that never end (see image). This is the only script it does this for. The script itself runs fine in the correct time and produces the correct data. It also looks like it finishes with a green tick on the code cell, but the spark job remains 'in progress'. The data from the MLV is accessible once the code block finishes, so no locks going on that I can see. It also doesn't look like these jobs are impacting our capacity usage that I can see either. All of the parent jobs seem to contain a child thats complete, but the parent stays in progress forever..
Anyone seen this before and know what they could be?
left long enough, the children jobs turn to this, but the timer on the parent keeps running
The only difference between ones we've done so far and this on in particular is that there are 2 CTE's in the SQL.. didnt think those were an issue
If I F5 the screen at any point after the green tick shows up in the code cell, it all goes away. Is this just weird GUI feeding back spark cleanup or somehting I can ignore or do I need to address something here?
Thanks
yeah this seems to be a UI bug. Open a ticket to report it is my suggestion.
If you found this helpful, consider giving some Kudos. If I answered your question or solved your problem, mark this post as the solution.
6 Replies
Hi TheFifthJoin,
Very interesting. This looks like a UI bug to me, especially if once you refresh it all goes away.
Though it is interesting that you're only seeing this UI bug on one notebook and not on all of them.
Is there anything special about the configuration of this specific notebook? Is it using a different spark pool or environment by chance?
If you found this helpful, consider giving some Kudos. If I answered your question or solved your problem, mark this post as the solution.
I have an MLV environment set up with some resource scripts, so all of them use that - no change there.
The only difference is the size of the script - getting into Gold so we're looking at bigger scripts to denormalize our structure, and some CTE's..
Everything else is the same (I use a template script for MLV creation so apart from the actual SQL script between the quotations in spark.sql(""), the rest is identical)
yeah this seems to be a UI bug. Open a ticket to report it is my suggestion.
If you found this helpful, consider giving some Kudos. If I answered your question or solved your problem, mark this post as the solution.
Hi TheFifthJoin ,
Interesting,
but does the notebook report “Succeeded” at some point in the pipeline, or does the notebook time out?As far as I understand, everything is running smoothly.
I would also suspect a GUI bug here. Personally, I haven't noticed this yet.I'll keep an eye out for it.
Best regards
it never returns 'Succeeded' on the Spark Job line. It'll sit there with the parent jobs on a running timer, but the child job under it will eventually time out and show 'unable to load stages' after about 20 minutes (picture 2 in the original post).
