Forum Discussion
Solved
Overwriting a csv using Spark notebook creates artifacts
Hi everyone, It might be a bit stupid of a question, but is there a way to overwrite csv files in a lakehouse (using Pyspark notebook) without creation of additional folders/artifacts? Right ...
Hi AndyDDC
Thank you for answering.
I had it saved directly to the lakehouse Files, without any subfolder. When I overwrite it, it lands in a subfolder that is named as the file before, while inside there are two entities:
- csv file with hashed name
- SUCCESS artifact
I see from your example that it works fine after it creates the aforementioned structure. That is useful knowledge. I guess I have to get accustomed to this structure, at the end of a day it is still human-readable.
Yes it's advisable to have sub-folders when writing, as there could be overwrite issues.
If my reply has been helpful please consider marking it as the solution.
Glad it's sorted now