topic Re: Anyone else having random errors show up in notebooks? in Data Engineering

Anyone else having random errors show up in notebooks?

arpost — Tue, 18 Mar 2025 14:19:23 GMT

Greetings, community. We recently started getting the following error with our notebooks:

Py4JJavaError: An error occurred while calling...java.lang.IllegalStateException: SparkSession should only be created and accessed on the driver.

Some form of the code in these notebooks has been running fine but is now throwing errors. I've tested the code in a new notebook all by itself and am getting the same error, which made me wonder if this means Microsoft changed something in the background.

Anyone else getting these errors all of a sudden?

Re: Anyone else having random errors show up in notebooks?

v-karpurapud — Wed, 19 Mar 2025 04:49:35 GMT

Hello arpost

Thank you for reaching out to Microsoft Community Forum.

The error message

Py4JJavaError: An error occurred while calling...java.lang.IllegalStateException: SparkSession should only be created and accessed on the driver usually indicates an attempt to create or access a SparkSession on a worker node instead of the driver node.

Ensure the SparkSession is created only once and on the driver node. Avoid creating it inside functions executed on worker nodes.
Verify that code involving SparkSession operations is not executed within transformations or actions on worker nodes, such as map, filter, or other RDD operations.
Ensure the SparkSession is not referenced as a global variable inside functions on worker nodes.

If the issue persists, consider sharing the specific code snippet causing the error for a detailed analysis.

If my response has resolved your query, please mark it as the Accepted Solution to assist others. A 'Kudos' would also be appreciated if you found my response helpful.

Thank you!

Re: Anyone else having random errors show up in notebooks?

v-karpurapud — Mon, 24 Mar 2025 06:32:59 GMT

Hi arpost

May I ask if you have resolved this issue? If so, please mark the helpful reply and accept it as the solution. This will be helpful for other community members who have similar problems to solve it faster.

Thank you.

Re: Anyone else having random errors show up in notebooks?

v-karpurapud — Sat, 29 Mar 2025 11:53:29 GMT

Hi arpost

I wanted to check if you had the opportunity to review the information provided. Please feel free to contact us if you have any further questions. If my response has addressed your query, please accept it as a solution and give a 'Kudos' so other members can easily find it.

Thank you.

Re: Anyone else having random errors show up in notebooks?

v-karpurapud — Tue, 01 Apr 2025 11:36:04 GMT

Hi arpost

I hope this information is helpful. Please let me know if you have any further questions or if you'd like to discuss this further. If this answers your question, please Accept it as a solution and give it a 'Kudos' so others can find it easily.

Thank you.