Power BI is turning 10! Tune in for a special live episode on July 24 with behind-the-scenes stories, product evolution highlights, and a sneak peek at what’s in store for the future.
Save the dateEnhance your career with this limited time 50% discount on Fabric and Power BI exams. Ends August 31st. Request your voucher.
Hi everyone,
It might be a bit stupid of a question, but is there a way to overwrite csv files in a lakehouse (using Pyspark notebook) without creation of additional folders/artifacts?
Right now when I use:
I can live with it, but it would be nice if it could just overwrite a file and leave it in the same destination.
Thank you in advance for any feedback.
Solved! Go to Solution.
Hi @Ostrzak what is the value in "file_path"? If I specify a Files folder to save the CSV to, it replaces the current CSV with a new version - changing the CSV filename in the process. But the old one has been removed.
Hi @Ostrzak what is the value in "file_path"? If I specify a Files folder to save the CSV to, it replaces the current CSV with a new version - changing the CSV filename in the process. But the old one has been removed.
Hi @AndyDDC
Thank you for answering.
I had it saved directly to the lakehouse Files, without any subfolder. When I overwrite it, it lands in a subfolder that is named as the file before, while inside there are two entities:
- csv file with hashed name
- SUCCESS artifact
I see from your example that it works fine after it creates the aforementioned structure. That is useful knowledge. I guess I have to get accustomed to this structure, at the end of a day it is still human-readable.
Hi @Ostrzak ,
We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. Otherwise, will respond back with the more details and we will try to help .
Yes it's advisable to have sub-folders when writing, as there could be overwrite issues.
If my reply has been helpful please consider marking it as the solution.
Glad it's sorted now