Earn a 50% discount on the DP-600 certification exam by completing the Fabric 30 Days to Learn It challenge.
Hi,
I seem to be getting a similiar error to the issue raised here.
I have a sample script where I am reading data from a table in the lakehouse - it worked fine initially so I am not sure what has changed but when I run the below code I get the following error. It seems to relate specifically to converting a spark df to Pandas - which works fine in fabric so not sure why its throwing an error in vs.
import pandas as pd
df=spark.read.load("specific_abfs_path")
df=df.toPandas()
Py4JJavaError Traceback (most recent call last)
Cell In[2], line 2
1 df=spark.read.load("specific_table_path")
----> 2 df=df.toPandas()
File c:\Users\AppData\Local\anaconda3\envs\fabric-synapse-runtime-1-2\lib\site-packages\pyspark\sql\pandas\conversion.py:208, in PandasConversionMixin.toPandas(self)
205 raise
207 # Below is toPandas without Arrow optimization.
--> 208 pdf = pd.DataFrame.from_records(self.collect(), columns=self.columns)
209 column_counter = Counter(self.columns)
211 corrected_dtypes: List[Optional[Type]] = [None] * len(self.schema)
File c:\Users\AppData\Local\anaconda3\envs\fabric-synapse-runtime-1-2\lib\site-packages\pyspark\sql\dataframe.py:1216, in DataFrame.collect(self)
1196 """Returns all the records as a list of :class:`Row`.
1197
1198 .. versionadded:: 1.3.0
(...)
1213 [Row(age=14, name='Tom'), Row(age=23, name='Alice'), Row(age=16, name='Bob')]
1214 """
1215 with SCCallSiteSync(self._sc):
-> 1216 sock_info = self._jdf.collectToPython()
1217 return list(_load_from_socket(sock_info, BatchedSerializer(CPickleSerializer())))
File c:\Users\AppData\Local\anaconda3\envs\fabric-synapse-runtime-1-2\lib\site-packages\py4j\java_gateway.py:1321, in JavaMember.__call__(self, *args)
...
at py4j.commands.CallCommand.execute(CallCommand.java:79)
at py4j.ClientServerConnection.waitForCommands(ClientServerConnection.java:182)
at py4j.ClientServerConnection.run(ClientServerConnection.java:106)
at java.lang.Thread.run(Thread.java:750)
Output is truncated. View as a scrollable element or open in a text editor. Adjust cell output settings...
Hi,
Yes a ticket has been raised for this issue. #2402220050002621. It is still in progress.
Thanks,
Hi @s_arah
Thanks for using Microsoft Fabric Community.
Apologies for the inconvenience that you are facing here.
Please refer to the document for prerequisites for the Synapse VS Code extension.
I found a thread which is similar to your ask, please refer to VS Code Synapse integration not working for more information this might help you.
If the issue still persists, please do let us know. Glad to help.
I hope this information helps.
Thanks.
Hi @s_arah
We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution please do share that same with the community as it can be helpful to others.
Otherwise, will respond back with the more details and we will try to help.
Thanks.
Hi,
I have attempted the suggestions in the post you shared but the issue is still ongoing.
Is there anything else that may be causing the issue?
I have included below the code + output
Hi @s_arah
Apologies for the delay in reponse from my end. Inorder to understand the exact issue, it requires a deeper investigation from our engineering team about your workspace and the logic behind it to properly understand what might be happening.
Please go ahead and raise a support ticket to reach our support team: support-ticket
Please provide the ticket number here as we can keep an eye on it.
Thanks.
Hi @s_arah
Following up on my previous inquiry, have you had the opportunity to raise a support ticket regarding this issue. If so, could you please provide the ticket number here as we can keep an eye on it.
Thanks.