Forum Discussion
VS Code notebook - error - When converting spark df to pandas
Hi,
I seem to be getting a similiar error to the issue raised here.
I have a sample script where I am reading data from a table in the lakehouse - it worked fine initially so I am not sure what has changed but when I run the below code I get the following error. It seems to relate specifically to converting a spark df to Pandas - which works fine in fabric so not sure why its throwing an error in vs.
import pandas as pd
df=spark.read.load("specific_abfs_path")
df=df.toPandas() Py4JJavaError Traceback (most recent call last)
Cell In[2], line 2
1 df=spark.read.load("specific_table_path")
----> 2 df=df.toPandas()
File c:\Users\AppData\Local\anaconda3\envs\fabric-synapse-runtime-1-2\lib\site-packages\pyspark\sql\pandas\conversion.py:208, in PandasConversionMixin.toPandas(self)
205 raise
207 # Below is toPandas without Arrow optimization.
--> 208 pdf = pd.DataFrame.from_records(self.collect(), columns=self.columns)
209 column_counter = Counter(self.columns)
211 corrected_dtypes: List[Optional[Type]] = [None] * len(self.schema)
File c:\Users\AppData\Local\anaconda3\envs\fabric-synapse-runtime-1-2\lib\site-packages\pyspark\sql\dataframe.py:1216, in DataFrame.collect(self)
1196 """Returns all the records as a list of :class:`Row`.
1197
1198 .. versionadded:: 1.3.0
(...)
1213 [Row(age=14, name='Tom'), Row(age=23, name='Alice'), Row(age=16, name='Bob')]
1214 """
1215 with SCCallSiteSync(self._sc):
-> 1216 sock_info = self._jdf.collectToPython()
1217 return list(_load_from_socket(sock_info, BatchedSerializer(CPickleSerializer())))
File c:\Users\AppData\Local\anaconda3\envs\fabric-synapse-runtime-1-2\lib\site-packages\py4j\java_gateway.py:1321, in JavaMember.__call__(self, *args)
...
at py4j.commands.CallCommand.execute(CallCommand.java:79)
at py4j.ClientServerConnection.waitForCommands(ClientServerConnection.java:182)
at py4j.ClientServerConnection.run(ClientServerConnection.java:106)
at java.lang.Thread.run(Thread.java:750)
Output is truncated. View as a scrollable element or open in a text editor. Adjust cell output settings...
6 Replies
Hi Anonymous
Thanks for using Microsoft Fabric Community.
Apologies for the inconvenience that you are facing here.
Please refer to the document for prerequisites for the Synapse VS Code extension.
I found a thread which is similar to your ask, please refer to VS Code Synapse integration not working for more information this might help you.
If the issue still persists, please do let us know. Glad to help.
I hope this information helps.
Thanks.
Hi Anonymous
We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution please do share that same with the community as it can be helpful to others.
Otherwise, will respond back with the more details and we will try to help.
Thanks.Hi,
I have attempted the suggestions in the post you shared but the issue is still ongoing.
- I have uninstalled and reinstalled vs code + synapse extension
- I have reinstalled pyspark + spark-lighter-lib
- I have tried searching for logs but it looks like the log files havent saved for a few days.
Is there anything else that may be causing the issue?
I have included below the code + output
