Forum Discussion
Pandas API on Spark
Hi frithjof_v ,
Using the sample data as an example, try writing the results back to the ABFS path, like this:
import pyspark.pandas as ps df = ps.read_parquet("abfss://[email protected]/LH_002.Lakehouse/Tables/publicholidays") df = df.head(10) df.to_csv('abfss://[email protected]/LH_002.Lakehouse/Files/sample_datasets/testname.csv', index=False)Here it recognizes testname.csv as a folder, I did some searching and it seems that you can't specify the generated filename under spark at the moment.
azure synapse - How to write .csv File in ADLS Using Pyspark - Stack Overflow
pyspark.pandas.read_parquet — PySpark 3.5.2 documentation (apache.org)
pyspark.pandas.DataFrame.to_csv — PySpark 3.5.2 documentation (apache.org)
Best Regards,
Gao
Community Support TeamIf there is any post helps, then please consider Accept it as the solution to help the other members find it more quickly.
If I misunderstand your needs or you still have problems on it, please feel free to let us know. Thanks a lot!How to get your questions answered quickly -- How to provide sample data in the Power BI Forum
Hi frithjof_v ,
Using the sample data as an example, try writing the results back to the ABFS path, like this:
import pyspark.pandas as ps
df = ps.read_parquet("abfss://[email protected]/LH_002.Lakehouse/Tables/publicholidays")
df = df.head(10)
df.to_csv('abfss://[email protected]/LH_002.Lakehouse/Files/sample_datasets/testname.csv', index=False)
Here it recognizes testname.csv as a folder, I did some searching and it seems that you can't specify the generated filename under spark at the moment.
azure synapse - How to write .csv File in ADLS Using Pyspark - Stack Overflow
pyspark.pandas.read_parquet — PySpark 3.5.2 documentation (apache.org)
pyspark.pandas.DataFrame.to_csv — PySpark 3.5.2 documentation (apache.org)
Best Regards,
Gao
Community Support Team
If there is any post helps, then please consider Accept it as the solution to help the other members find it more quickly.
If I misunderstand your needs or you still have problems on it, please feel free to let us know. Thanks a lot!
How to get your questions answered quickly -- How to provide sample data in the Power BI Forum
Thank you Anonymous ,
So in order to specify the name of a single csv file, I guess we cannot use Pandas API on Spark?
Instead we would need to use regular Pandas https://learn.microsoft.com/en-us/fabric/data-science/read-write-pandas or Polars. I have tested regular Pandas and Polars, both are able to write single csv files with specific file name.
I was just curious if we could achieve the same with Pandas API on Spark. However it seems we need to use regular Pandas or Polars to achieve this. Thanks!