VS Code notebook - error - When converting spark df to pandas

[Anonymous]

Hi, 

I seem to be getting a similiar error to the issue raised here.

I have a sample script where I am reading data from a table in the lakehouse - it worked fine initially so I am not sure what has changed but when I run the below code I get the following error. It seems to relate specifically to converting a spark df to Pandas - which works fine in fabric so not sure why its throwing an error in vs.

import pandas as pd
df=spark.read.load("specific_abfs_path")
df=df.toPandas() 

Py4JJavaError                             Traceback (most recent call last)
Cell In[2], line 2
      1 df=spark.read.load("specific_table_path")
----> 2 df=df.toPandas() 

File c:\Users\AppData\Local\anaconda3\envs\fabric-synapse-runtime-1-2\lib\site-packages\pyspark\sql\pandas\conversion.py:208, in PandasConversionMixin.toPandas(self)
    205             raise
    207 # Below is toPandas without Arrow optimization.
--> 208 pdf = pd.DataFrame.from_records(self.collect(), columns=self.columns)
    209 column_counter = Counter(self.columns)
    211 corrected_dtypes: List[Optional[Type]] = [None] * len(self.schema)

File c:\Users\AppData\Local\anaconda3\envs\fabric-synapse-runtime-1-2\lib\site-packages\pyspark\sql\dataframe.py:1216, in DataFrame.collect(self)
   1196 """Returns all the records as a list of :class:`Row`.
   1197 
   1198 .. versionadded:: 1.3.0
   (...)
   1213 [Row(age=14, name='Tom'), Row(age=23, name='Alice'), Row(age=16, name='Bob')]
   1214 """
   1215 with SCCallSiteSync(self._sc):
-> 1216     sock_info = self._jdf.collectToPython()
   1217 return list(_load_from_socket(sock_info, BatchedSerializer(CPickleSerializer())))

File c:\Users\AppData\Local\anaconda3\envs\fabric-synapse-runtime-1-2\lib\site-packages\py4j\java_gateway.py:1321, in JavaMember.__call__(self, *args)
...
	at py4j.commands.CallCommand.execute(CallCommand.java:79)
	at py4j.ClientServerConnection.waitForCommands(ClientServerConnection.java:182)
	at py4j.ClientServerConnection.run(ClientServerConnection.java:106)
	at java.lang.Thread.run(Thread.java:750)
Output is truncated. View as a scrollable element or open in a text editor. Adjust cell output settings...

[Anonymous]

Hi, 

Yes a ticket has been raised for this issue. #2402220050002621. It is still in progress.

Thanks,

[v-cboorla-msft]

Hi @Anonymous 

Thanks for using Microsoft Fabric Community.

Apologies for the inconvenience that you are facing here.

Please refer to the document for prerequisites for the Synapse VS Code extension.

I found a thread which is similar to your ask, please refer to VS Code Synapse integration not working for more information this might help you.

If the issue still persists, please do let us know. Glad to help.

I hope this information helps.

Thanks.

[v-cboorla-msft]

Hi @Anonymous 

We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution please do share that same with the community as it can be helpful to others.
Otherwise, will respond back with the more details and we will try to help.

Thanks.

[Anonymous]

Hi,

I have attempted the suggestions in the post you shared but the issue is still ongoing. 

• I have uninstalled and reinstalled vs code + synapse extension 
• I have reinstalled pyspark + spark-lighter-lib
• I have tried searching for logs but it looks like the log files havent saved for a few days. 

Is there anything else that may be causing the issue? 

I have included below the code + output 

[v-cboorla-msft]

Hi @Anonymous 

Apologies for the delay in reponse from my end. Inorder to understand the exact issue, it requires a deeper investigation from our engineering team about your workspace and the logic behind it to properly understand what might be happening. 

Please go ahead and raise a support ticket to reach our support team: support-ticket 

Please provide the ticket number here as we can keep an eye on it.

Thanks.

[v-cboorla-msft]

Hi @Anonymous 

Following up on my previous inquiry, have you had the opportunity to raise a support ticket regarding this issue. If so, could you please provide the ticket number here as we can keep an eye on it.

Thanks.