Forum Discussion

ronan_b6's avatar
ronan_b6
Advocate II
2 years ago
Solved

Issues running Spark Notebook

Hi I'm runing spark notebooks on Fabric trial, it's been working great and no time out issues, or severe dalys in sessions starting.   But today, notebooks do not start the spark session, returning...
Notebook
  • ronan_b6's avatar
    ronan_b6
    2 years ago

    This issue persisted for a week. I updated to runtime 1.3 at the time and it had no effect on the issue. MS stated at the time it was a resource allocation issue. Since 07-18-2024 I have not experienced this issue with my pipelines. For now it seems MS has resolved the issue. (Central Canada)

PhantomThread's avatar
PhantomThread
Advocate II
2 years ago

I've just tried the 1.3 change and got a Pandas error. Further reading this morning suggests I'm hitting memory issues based on the size of the data frame I'm outputting... Have edited this post as my code is the issue here, not Fabric 🙂

PhantomThread's avatar
PhantomThread
Advocate II
2 years ago

So just to reply to myself... I've had other pipelines calling notebooks over the weekend - default is now 1.3 - and they've worked ok (and have calls to pandas to output files).

 

I've not yet reverted to 1.2 to test it, but can confirm that 1.3 is working.

 

Looks like something strange is going on with this particular notebook...