Forum Discussion
Pandas API on Spark
Hi all,
I want to test how to use Pandas API on Spark.
I'm reading a delta table and I want to save it to a single CSV file.
I already know how to do this in regular pandas, but I'd like to try it with the Pandas API on Spark.
I have a default lakehouse attached to my Notebook, and it works if I am using regular pandas.
import pyspark.pandas as ps
df = spark.read.load("abfss://[email protected]/TestLakehouse.Lakehouse/Tables/RandomNumbers")
df = df.limit(100)
df_pandasOnSpark = df.pandas_api()
df_pandasOnSpark.to_csv('/lakehouse/default/Files/RandomNumbers_PandasOnSpark.csv', header=True, index = False)
I'm getting the following error:
Py4JJavaError: An error occurred while calling o6290.save. : Operation failed: "Bad Request", 400, HEAD, http://onelake.dfs.fabric.microsoft.com/<workspaceGUID>/lakehouse/default/Files/RandomNumbers_PandasOnSpark.csv?upn=false&action=getStatus&timeout=90
Anyone knows what I'm doing wrong & how I could get this to work?
Thanks in advance!
Hi frithjof_v ,
Using the sample data as an example, try writing the results back to the ABFS path, like this:
import pyspark.pandas as ps df = ps.read_parquet("abfss://[email protected]/LH_002.Lakehouse/Tables/publicholidays") df = df.head(10) df.to_csv('abfss://[email protected]/LH_002.Lakehouse/Files/sample_datasets/testname.csv', index=False)Here it recognizes testname.csv as a folder, I did some searching and it seems that you can't specify the generated filename under spark at the moment.
azure synapse - How to write .csv File in ADLS Using Pyspark - Stack Overflow
pyspark.pandas.read_parquet — PySpark 3.5.2 documentation (apache.org)
pyspark.pandas.DataFrame.to_csv — PySpark 3.5.2 documentation (apache.org)
Best Regards,
Gao
Community Support TeamIf there is any post helps, then please consider Accept it as the solution to help the other members find it more quickly.
If I misunderstand your needs or you still have problems on it, please feel free to let us know. Thanks a lot!How to get your questions answered quickly -- How to provide sample data in the Power BI Forum
2 Replies
Hi frithjof_v ,
Using the sample data as an example, try writing the results back to the ABFS path, like this:
import pyspark.pandas as ps df = ps.read_parquet("abfss://[email protected]/LH_002.Lakehouse/Tables/publicholidays") df = df.head(10) df.to_csv('abfss://[email protected]/LH_002.Lakehouse/Files/sample_datasets/testname.csv', index=False)Here it recognizes testname.csv as a folder, I did some searching and it seems that you can't specify the generated filename under spark at the moment.
azure synapse - How to write .csv File in ADLS Using Pyspark - Stack Overflow
pyspark.pandas.read_parquet — PySpark 3.5.2 documentation (apache.org)
pyspark.pandas.DataFrame.to_csv — PySpark 3.5.2 documentation (apache.org)
Best Regards,
Gao
Community Support TeamIf there is any post helps, then please consider Accept it as the solution to help the other members find it more quickly.
If I misunderstand your needs or you still have problems on it, please feel free to let us know. Thanks a lot!How to get your questions answered quickly -- How to provide sample data in the Power BI Forum
Thank you Anonymous ,
So in order to specify the name of a single csv file, I guess we cannot use Pandas API on Spark?
Instead we would need to use regular Pandas https://learn.microsoft.com/en-us/fabric/data-science/read-write-pandas or Polars. I have tested regular Pandas and Polars, both are able to write single csv files with specific file name.
I was just curious if we could achieve the same with Pandas API on Spark. However it seems we need to use regular Pandas or Polars to achieve this. Thanks!
