Forum Discussion
Dataflow Gen2 - Excel in SharePoint to Lakehouse - WriteToDataDestination Error
Hi JasonArmstrong , Thank you for reaching out to the Microsoft Community Forum.
The issue happens when a Dataflow writes to a Lakehouse table that already exists from a previous run. Even though you've set the update method to Replace, Fabric doesn't actually delete the physical OneLake path, it just tries to overwrite the files. If that path hasn't been fully released yet, which often occurs when Excel files are being ingested through an On-Prem Gateway, you’ll hit the “path already exists” error.
When you write to a staging table instead, you’re usually creating a fresh, isolated target that isn’t reused or referenced elsewhere. That makes it far less likely to be locked or conflicted. And after the Dataflow finishes, you use a Spark notebook to drop and recreate the final table. Spark handles that clean-up at the storage level properly removing the folder and all contents in a way Dataflow doesn’t.
You asked whether you could just drop the final table before running the Dataflow. Technically yes, but in practice it’s unreliable. OneLake can hold onto the storage path momentarily even after deletion and if the Dataflow tries to write before it’s fully cleared, the conflict returns. That’s why the staging pattern works better, the Dataflow runs without touching production and Spark takes care of the overwrite safely afterward. It’s not just moving the problem, it’s solving it by separating responsibilities between tools that handle them differently.
If this helped solve the issue, please consider marking it “Accept as Solution” so others with similar queries may find it more easily. If not, please share the details, always happy to help.
Thank you.
Community SupportHi JasonArmstrong , Thank you for reaching out to the Microsoft Community Forum.
This happens maybe because the underlying Lakehouse storage path wasn’t properly cleared from a previous run. It’s common when using an On-Premises Gateway with Excel files from SharePoint, where cleanup during a Replace operation can be inconsistent.
The most reliable fix is to avoid writing directly to your final table. Instead, write the Dataflow output to a temporary staging table. Then use a notebook or pipeline step to drop and recreate the final table before copying data from the staging table. Here’s how to drop the existing table in a notebook:
spark.sql("DROP TABLE IF EXISTS lakehouse.SP_Excel_ad_narrative")
This ensures the path is cleared before new data is written, preventing the overwrite conflict. Stick with Fixed Schema and explicitly defined data types, they’re still the most stable option when working with Excel inputs.
If your issue persists, then the best next step is to report this issue to Microsoft Support, as it may be a bug or a backend issue. You can provide them with all the troubleshooting steps you've already taken along with error screenshots and other necessary details, which will help them understand the issue better and provide a resolution. They might be able to provide a solution that isn't immediately obvious.
Below is the link to help create Microsoft Support ticket:
How to create a Fabric and Power BI Support ticket - Power BI | Microsoft Learn
If this helped solve the issue, please consider marking it “Accept as Solution” so others with similar queries may find it more easily. If not, please share the details, always happy to help.
Thank you.
Thank you for the reply - I will test this and get back to you to confirm.
- v-hashadapu
Hi JasonArmstrong , Thanks for the update, hope it works for you!
Hi v-hashadapu
Thank you, I ended up creating a pipeline which runs a notebook to drop the tables from the lakehouse before executing the dataflow to ingest them which solved the issue.
"Dataflow writes to a Lakehouse table that already exists from a previous run. Even though you've set the update method to Replace, Fabric doesn't actually delete the physical OneLake path, it just tries to overwrite the files. If that path hasn't been fully released yet, which often occurs when Excel files are being ingested through an On-Prem Gateway, you’ll hit the “path already exists” error."
As for your point about using a staging table. I am unsure to do that in a way that is materially different to writing to the lakehouse. Is there a specific type of storage that is a staging table I would need to create? In my mind if i write the dataflow to a different table before rerouting it to the final lakehouse table will it not encounter the same issue with the path to the staging table?
I am relatively new to using Fabric and am aware of the concept of staging but not sure how to create a specific staging table. Are these lakehouse tables or something seperate.
Hopefully this gap can be fixed in future iterations of Fabric as to me it seems like something which should be standard functionality.
Thanks again!