Pandas API on Spark

[frithjof_v]

Hi all,

I want to test how to use Pandas API on Spark.

I'm reading a delta table and I want to save it to a single CSV file.

I already know how to do this in regular pandas, but I'd like to try it with the Pandas API on Spark.

I have a default lakehouse attached to my Notebook, and it works if I am using regular pandas.

import pyspark.pandas as ps

df = spark.read.load("abfss://Fabric@onelake.dfs.fabric.microsoft.com/TestLakehouse.Lakehouse/Tables/RandomNumbers")
df = df.limit(100)

df_pandasOnSpark = df.pandas_api()

df_pandasOnSpark.to_csv('/lakehouse/default/Files/RandomNumbers_PandasOnSpark.csv', header=True, index = False)

I'm getting the following error:

Py4JJavaError: An error occurred while calling o6290.save. : Operation failed: "Bad Request", 400, HEAD, http://onelake.dfs.fabric.microsoft.com/<workspaceGUID>/lakehouse/default/Files/RandomNumbers_PandasOnSpark.csv?upn=false&action=getStatus&timeout=90   

Anyone knows what I'm doing wrong & how I could get this to work?

Thanks in advance!

[Anonymous · Accepted solution]

Hi @frithjof_v ,

Using the sample data as an example, try writing the results back to the ABFS path, like this:

import pyspark.pandas as ps


df = ps.read_parquet("abfss://xxxxxx@onelake.dfs.fabric.microsoft.com/LH_002.Lakehouse/Tables/publicholidays")

df = df.head(10)

df.to_csv('abfss://xxxxxx@onelake.dfs.fabric.microsoft.com/LH_002.Lakehouse/Files/sample_datasets/testname.csv', index=False)

Here it recognizes testname.csv as a folder, I did some searching and it seems that you can't specify the generated filename under spark at the moment.

azure synapse - How to write .csv File in ADLS Using Pyspark - Stack Overflow

pyspark.pandas.read_parquet — PySpark 3.5.2 documentation (apache.org)

pyspark.pandas.DataFrame.to_csv — PySpark 3.5.2 documentation (apache.org)

Best Regards,
Gao
Community Support Team

If there is any post helps, then please consider Accept it as the solution  to help the other members find it more quickly.
If I misunderstand your needs or you still have problems on it, please feel free to let us know. Thanks a lot!

How to get your questions answered quickly --  How to provide sample data in the Power BI Forum

[frithjof_v]

Thank you @Anonymous ,

So in order to specify the name of a single csv file, I guess we cannot use Pandas API on Spark?

Instead we would need to use regular Pandas https://learn.microsoft.com/en-us/fabric/data-science/read-write-pandas or Polars. I have tested regular Pandas and Polars, both are able to write single csv files with specific file name. 

I was just curious if we could achieve the same with Pandas API on Spark. However it seems we need to use regular Pandas or Polars to achieve this. Thanks!