Forum Discussion

Ostrzak's avatar
Ostrzak
Helper II
2 years ago
Solved

Overwriting a csv using Spark notebook creates artifacts

Hi everyone,

 

It might be a bit stupid of a question, but is there a way to overwrite csv files in a lakehouse (using Pyspark notebook) without creation of additional folders/artifacts?

 

Right now when I use:

df.write.mode("overwrite").csv("file_path")
 
A new subfolder is created (named as original file), within it SUCCESS artifact and a .csv file with a hashed name. 

I can live with it, but it would be nice if it could just overwrite a file and leave it in the same destination. 

Thank you in advance for any feedback.

data engineering
  • AndyDDC's avatar
    AndyDDC
    2 years ago

    Hi Ostrzak what is the value in "file_path"?  If I specify a Files folder to save the CSV to, it replaces the current CSV with a new version - changing the CSV filename in the process.  But the old one has been removed.

     

     

     

     

4 Replies

  • AndyDDC's avatar
    AndyDDC
    Most Valuable Professional
    2 years ago

    Hi Ostrzak what is the value in "file_path"?  If I specify a Files folder to save the CSV to, it replaces the current CSV with a new version - changing the CSV filename in the process.  But the old one has been removed.

     

     

     

     

    • Ostrzak's avatar
      Ostrzak
      Helper II
      2 years ago

      Hi AndyDDC 

       

      Thank you for answering.

      I had it saved directly to the lakehouse Files, without any subfolder. When I overwrite it, it lands in a subfolder that is named as the file before, while inside there are  two entities:

      - csv file with hashed name

      - SUCCESS artifact

      I see from your example that it works fine after it creates the aforementioned structure. That is useful knowledge. I guess I have to get accustomed to this structure, at the end of a day it is still human-readable.

      • AndyDDC's avatar
        AndyDDC
        Most Valuable Professional
        2 years ago

        Yes it's advisable to have sub-folders when writing, as there could be overwrite issues.

        If my reply has been helpful please consider marking it as the solution.

        Glad it's sorted now