Forum Discussion
Issues running Spark Notebook
This issue persisted for a week. I updated to runtime 1.3 at the time and it had no effect on the issue. MS stated at the time it was a resource allocation issue. Since 07-18-2024 I have not experienced this issue with my pipelines. For now it seems MS has resolved the issue. (Central Canada)
Hey everyone,
I just had some resolution on this issue by enabling runtime 1.3 (Preview) on my workspace. I have restarted a pipeline with hundreds of Notebook executions and they all seem to be running smoothly with the newer (Preview) runtime.
Of course this could just be a convenient glitch, so I will comment back tomorrow on the progress, but it looks promising to me as I was able to reproduce the issue with a manual execution of a notebook, and that has also been working since I made the config change.
If you are able to reproduce the issue while running Notebooks manually, I recommend experimenting with a custom Notebook environment set to runtime 1.3 and see if you get any better results.
I will follow up tomorrow with a status update.
Thank you
I've just tried the 1.3 change and got a Pandas error. Further reading this morning suggests I'm hitting memory issues based on the size of the data frame I'm outputting... Have edited this post as my code is the issue here, not Fabric 🙂
So just to reply to myself... I've had other pipelines calling notebooks over the weekend - default is now 1.3 - and they've worked ok (and have calls to pandas to output files).
I've not yet reverted to 1.2 to test it, but can confirm that 1.3 is working.
Looks like something strange is going on with this particular notebook...
FYI: I got an update from MS last friday that this issue should be resolved. So we should be able to revert back to runtime 1.2.
I have not tested this yet, but just passing on the message.