Forum Discussion

Jeanxyz's avatar
Jeanxyz
Power Participant
1 year ago
Solved

how to write a spark dataframe into csv file without losing column headers

I have created a notebook and a spark data frame, when I write the data frame into a csv file, the column header is not written into csv. Is that normal? *** yearlySales = df.select(year(col("Orde...
data engineering
  • nilendraFabric's avatar
    nilendraFabric
    1 year ago

    Hi Jeanxyz 

     

    yearlySales.write.mode("overwrite").option("header", True).format("csv").save("Files/yearlySales.csv")

     

    option(“header”, True): Ensures that the column headers are written as the first row in the CSV output.

     

v-saisrao-msft's avatar
v-saisrao-msft
Community Support
1 year ago

Hi Jeanxyz,

Thank you for reaching out to the Microsoft Fabric Forum Community. 

Check the solution provided by nilendraFabric, adding.option("header", True) to your .write() operation will ensure the column headers are written as the first row in your CSV file. 

You can also refer to the official Apache Spark documentation for more context on the header option: 

 

CSV Files - Spark 3.3.2 Documentation 

 

If this post helps, then please give us ‘Kudos’ and consider Accept it as a solution to help the other members find it more quickly.

 

Thank you.