Forum Discussion
Solved
Issues running Spark Notebook
Hi I'm runing spark notebooks on Fabric trial, it's been working great and no time out issues, or severe dalys in sessions starting. But today, notebooks do not start the spark session, returning...
This issue persisted for a week. I updated to runtime 1.3 at the time and it had no effect on the issue. MS stated at the time it was a resource allocation issue. Since 07-18-2024 I have not experienced this issue with my pipelines. For now it seems MS has resolved the issue. (Central Canada)
I've just tried the 1.3 change and got a Pandas error. Further reading this morning suggests I'm hitting memory issues based on the size of the data frame I'm outputting... Have edited this post as my code is the issue here, not Fabric 🙂
FYI: I got an update from MS last friday that this issue should be resolved. So we should be able to revert back to runtime 1.2.
I have not tested this yet, but just passing on the message.