Earn a 50% discount on the DP-600 certification exam by completing the Fabric 30 Days to Learn It challenge.
Hi everyone,
It might be a bit stupid of a question, but is there a way to overwrite csv files in a lakehouse (using Pyspark notebook) without creation of additional folders/artifacts?
Right now when I use:
I can live with it, but it would be nice if it could just overwrite a file and leave it in the same destination.
Thank you in advance for any feedback.
Solved! Go to Solution.
Hi @Ostrzak what is the value in "file_path"? If I specify a Files folder to save the CSV to, it replaces the current CSV with a new version - changing the CSV filename in the process. But the old one has been removed.
Hi @Ostrzak what is the value in "file_path"? If I specify a Files folder to save the CSV to, it replaces the current CSV with a new version - changing the CSV filename in the process. But the old one has been removed.
Hi @AndyDDC
Thank you for answering.
I had it saved directly to the lakehouse Files, without any subfolder. When I overwrite it, it lands in a subfolder that is named as the file before, while inside there are two entities:
- csv file with hashed name
- SUCCESS artifact
I see from your example that it works fine after it creates the aforementioned structure. That is useful knowledge. I guess I have to get accustomed to this structure, at the end of a day it is still human-readable.
Hi @Ostrzak ,
We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. Otherwise, will respond back with the more details and we will try to help .
Yes it's advisable to have sub-folders when writing, as there could be overwrite issues.
If my reply has been helpful please consider marking it as the solution.
Glad it's sorted now
Check out the April 2024 Fabric update to learn about new features.
Covering the world! 9:00-10:30 AM Sydney, 4:00-5:30 PM CET (Paris/Berlin), 7:00-8:30 PM Mexico City