topic Re: Overwriting a csv using Spark notebook creates artifacts in Data Engineering

Overwriting a csv using Spark notebook creates artifacts

Ostrzak — Mon, 30 Oct 2023 17:43:51 GMT

Hi everyone,

It might be a bit stupid of a question, but is there a way to overwrite csv files in a lakehouse (using Pyspark notebook) without creation of additional folders/artifacts?

Right now when I use:

df.write.mode("overwrite").csv("file_path")

A new subfolder is created (named as original file), within it SUCCESS artifact and a .csv file with a hashed name.

I can live with it, but it would be nice if it could just overwrite a file and leave it in the same destination.

Thank you in advance for any feedback.

Re: Overwriting a csv using Spark notebook creates artifacts

AndyDDC — Mon, 30 Oct 2023 19:33:53 GMT

Hi @Ostrzak what is the value in "file_path"? If I specify a Files folder to save the CSV to, it replaces the current CSV with a new version - changing the CSV filename in the process. But the old one has been removed.

Re: Overwriting a csv using Spark notebook creates artifacts

Ostrzak — Tue, 31 Oct 2023 11:13:30 GMT

Hi @AndyDDC

Thank you for answering.

I had it saved directly to the lakehouse Files, without any subfolder. When I overwrite it, it lands in a subfolder that is named as the file before, while inside there are two entities:

- csv file with hashed name

- SUCCESS artifact

I see from your example that it works fine after it creates the aforementioned structure. That is useful knowledge. I guess I have to get accustomed to this structure, at the end of a day it is still human-readable.

Re: Overwriting a csv using Spark notebook creates artifacts

AndyDDC — Tue, 31 Oct 2023 11:25:08 GMT

Yes it's advisable to have sub-folders when writing, as there could be overwrite issues.

If my reply has been helpful please consider marking it as the solution.

Glad it's sorted now

Re: Overwriting a csv using Spark notebook creates artifacts

Anonymous — Thu, 02 Nov 2023 08:38:02 GMT

Hi @Ostrzak ,

We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. Otherwise, will respond back with the more details and we will try to help .