topic how to write a spark dataframe into csv file without losing column headers in Data Engineering

how to write a spark dataframe into csv file without losing column headers

Jeanxyz — Sun, 25 May 2025 19:40:57 GMT

I have created a notebook and a spark data frame, when I write the data frame into a csv file, the column header is not written into csv. Is that normal?

***

yearlySales = df.select(year(col("OrderDate")).alias("Year")).groupBy("Year").count().orderBy("Year") yearlySales.write.mode("overwrite").format("csv").save("Files/yearlySales.csv")

***

Re: how to write a spark dataframe into csv file without losing column headers

nilendraFabric — Sun, 25 May 2025 21:54:36 GMT

Hi Jeanxyz

yearlySales.write.mode("overwrite").option("header", True).format("csv").save("Files/yearlySales.csv")

option(“header”, True): Ensures that the column headers are written as the first row in the CSV output.

Re: how to write a spark dataframe into csv file without losing column headers

Jeanxyz — Mon, 26 May 2025 08:04:26 GMT

Thanks a lot, nilendraFabric . This works. I have yet another question. What's the best Spark tutorial online? I searched on https://spark.apache.org/, but the documentation about dataframe write method seems quite limited.

Re: how to write a spark dataframe into csv file without losing column headers

v-saisrao-msft — Mon, 26 May 2025 08:41:11 GMT

Hi Jeanxyz,

Thank you for reaching out to the Microsoft Fabric Forum Community.

Check the solution provided by nilendraFabric, adding.option("header", True) to your .write() operation will ensure the column headers are written as the first row in your CSV file.

You can also refer to the official Apache Spark documentation for more context on the header option:

CSV Files - Spark 3.3.2 Documentation

If this post helps, then please give us ‘Kudos’ and consider Accept it as a solution to help the other members find it more quickly.

Thank you.