Anyone else having random errors show up in notebooks?

[arpost]

Greetings, community. We recently started getting the following error with our notebooks:

Py4JJavaError: An error occurred while calling...java.lang.IllegalStateException: SparkSession should only be created and accessed on the driver.

Some form of the code in these notebooks has been running fine but is now throwing errors. I've tested the code in a new notebook all by itself and am getting the same error, which made me wonder if this means Microsoft changed something in the background.

Anyone else getting these errors all of a sudden?

[v-karpurapud]

Hi @arpost 

I hope this information is helpful. Please let me know if you have any further questions or if you'd like to discuss this further. If this answers your question, please Accept it as a solution and give it a 'Kudos' so others can find it easily.

Thank you.

[v-karpurapud]

Hi @arpost 

I wanted to check if you had the opportunity to review the information provided. Please feel free to contact us if you have any further questions. If my response has addressed your query, please accept it as a solution and give a 'Kudos' so other members can easily find it.

Thank you.

[v-karpurapud]

Hi @arpost 

May I ask if you have resolved this issue? If so, please mark the helpful reply and accept it as the solution. This will be helpful for other community members who have similar problems to solve it faster.

Thank you.

[v-karpurapud · Accepted solution]

Hello @arpost 

Thank you for reaching out to Microsoft Community Forum.

The error message

Py4JJavaError: An error occurred while calling...java.lang.IllegalStateException: SparkSession should only be created and accessed on the driver usually indicates an attempt to create or access a SparkSession on a worker node instead of the driver node.

1. Ensure the SparkSession is created only once and on the driver node. Avoid creating it inside functions executed on worker nodes.
   
   
2. Verify that code involving SparkSession operations is not executed within transformations or actions on worker nodes, such as map, filter, or other RDD operations.
   
   
3. Ensure the SparkSession is not referenced as a global variable inside functions on worker nodes.
   

If the issue persists, consider sharing the specific code snippet causing the error for a detailed analysis.

If my response has resolved your query, please mark it as the Accepted Solution to assist others. A 'Kudos' would also be appreciated if you found my response helpful.

Thank you!