how to write a spark dataframe into csv file without losing column headers

[Jeanxyz]

I have created a notebook and a spark data frame, when I write the data frame into a csv file, the column header is not written into csv. Is that normal?

***

yearlySales = df.select(year(col("OrderDate")).alias("Year")).groupBy("Year").count().orderBy("Year")

yearlySales.write.mode("overwrite").format("csv").save("Files/yearlySales.csv")

***

[v-saisrao-msft]

Hi @Jeanxyz,

Thank you for reaching out to the Microsoft Fabric Forum Community. 

Check the solution provided by @nilendraFabric, adding.option("header", True) to your .write() operation will ensure the column headers are written as the first row in your CSV file. 

You can also refer to the official Apache Spark documentation for more context on the header option: 

CSV Files - Spark 3.3.2 Documentation 

If this post helps, then please give us ‘Kudos’ and consider Accept it as a solution to help the other members find it more quickly.

Thank you. 

[Jeanxyz]

Thanks a lot, @nilendraFabric . This works. I have yet another question. What's the best Spark tutorial online? I searched on https://spark.apache.org/, but the documentation about dataframe write method seems quite limited.

[nilendraFabric · Accepted solution]

Hi @Jeanxyz 

yearlySales.write.mode("overwrite").option("header", True).format("csv").save("Files/yearlySales.csv")

option(“header”, True): Ensures that the column headers are written as the first row in the CSV output.