Forum Discussion

TheFifthJoin's avatar
TheFifthJoin
Advocate I
9 months ago
Solved

Spark job never ends

We have some SQL code in a pyspark notebook creating a materialised lake view.    We do this a lot for our medallion architecture but for some reason, this script causes the spark session to create...
data engineering
notebook
Spark Job Definition
  • tayloramy's avatar
    tayloramy
    9 months ago

    yeah this seems to be a UI bug. Open a ticket to report it is my suggestion. 

     

    If you found this helpful, consider giving some Kudos. If I answered your question or solved your problem, mark this post as the solution.

spaceman127's avatar
spaceman127
Super User
9 months ago

Hi TheFifthJoin ,

 

Interesting,
but does the notebook report “Succeeded” at some point in the pipeline, or does the notebook time out?

As far as I understand, everything is running smoothly.
I would also suspect a GUI bug here. Personally, I haven't noticed this yet.

I'll keep an eye out for it.

 

Best regards

TheFifthJoin's avatar
TheFifthJoin
Advocate I
9 months ago

it never returns 'Succeeded' on the Spark Job line. It'll sit there with the parent jobs on a running timer, but the child job under it will eventually time out and show 'unable to load stages' after about 20 minutes (picture 2 in the original post).