Forum Discussion

Ostrzak's avatar
Ostrzak
Helper II
2 years ago
Solved

Overwriting a csv using Spark notebook creates artifacts

Hi everyone,   It might be a bit stupid of a question, but is there a way to overwrite csv files in a lakehouse (using Pyspark notebook) without creation of additional folders/artifacts?   Right ...
data engineering
  • AndyDDC's avatar
    AndyDDC
    2 years ago

    Hi Ostrzak what is the value in "file_path"?  If I specify a Files folder to save the CSV to, it replaces the current CSV with a new version - changing the CSV filename in the process.  But the old one has been removed.

     

     

     

     

Ostrzak's avatar
Ostrzak
Helper II
2 years ago

Hi AndyDDC 

 

Thank you for answering.

I had it saved directly to the lakehouse Files, without any subfolder. When I overwrite it, it lands in a subfolder that is named as the file before, while inside there are  two entities:

- csv file with hashed name

- SUCCESS artifact

I see from your example that it works fine after it creates the aforementioned structure. That is useful knowledge. I guess I have to get accustomed to this structure, at the end of a day it is still human-readable.

Anonymous's avatar
Anonymous
Not applicable
2 years ago

Hi Ostrzak ,

We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. Otherwise, will respond back with the more details and we will try to help .