Forum Discussion
Issues running Spark Notebook
This issue persisted for a week. I updated to runtime 1.3 at the time and it had no effect on the issue. MS stated at the time it was a resource allocation issue. Since 07-18-2024 I have not experienced this issue with my pipelines. For now it seems MS has resolved the issue. (Central Canada)
Hey everyone,
I just had some resolution on this issue by enabling runtime 1.3 (Preview) on my workspace. I have restarted a pipeline with hundreds of Notebook executions and they all seem to be running smoothly with the newer (Preview) runtime.
Of course this could just be a convenient glitch, so I will comment back tomorrow on the progress, but it looks promising to me as I was able to reproduce the issue with a manual execution of a notebook, and that has also been working since I made the config change.
If you are able to reproduce the issue while running Notebooks manually, I recommend experimenting with a custom Notebook environment set to runtime 1.3 and see if you get any better results.
I will follow up tomorrow with a status update.
Thank you
i also have the same solution applied and data pipeline now works successfully on 1.3 spark.