topic Re: Fabric Dataflow Gen2 Lakehouse: Switch from "new" to "existing" table automatically in Dataflow

Fabric Dataflow Gen2 Lakehouse: Switch from "new" to "existing" table automatically

Anonymous — Tue, 30 Apr 2024 15:13:20 GMT

If I create a dataflow gen2 for a new bronze tier table, of course I choose the "new table" option as destination. I publish the dataflow successfully and the table henceforth exists. The dataflow therefore fails upon refresh, with an error along the lines of "table already exists". While this behavior is logical, I find it ridiculous. Because now I have to go through the whole dataflow destination settings again. And if I ever were to delete the table to start a fresh loading cycle, I would have to go through the destination settings yet again. TWICE! This gets ever more tedious, the more dataflows I amass in my workspace.

So I ask: Is there a setting that I'm missing or any solution for my problem that doesn't mean not using dataflows at all?

Thanks for any help!

Re: Fabric Dataflow Gen2 Lakehouse: Switch from "new" to "existing" table automatically

Anonymous — Thu, 02 May 2024 10:30:13 GMT

Hi Anonymous
Thanks for using Fabric Community.
The dataflow shouldn't fail just because the table exists. New table means the table will be created if it doesn't exist, otherwise the existing table will be used as target. Can you share more details around the failure?
Please send me the screenshot of the error along with the session id.

Thanks

Re: Fabric Dataflow Gen2 Lakehouse: Switch from "new" to "existing" table automatically

Anonymous — Fri, 03 May 2024 10:22:25 GMT

Interestingly, today several dataflows, which contain only one query, work as I would wish. The big one with about 30 queries still doesn't, though. Here's the error:

Re: Fabric Dataflow Gen2 Lakehouse: Switch from "new" to "existing" table automatically

Anonymous — Fri, 03 May 2024 10:52:35 GMT

Hi Anonymous
Thanks for the details. As per the error, I think you are not using the latest version of Gateway. This can also lead to errors sometimes. Can you please download the latest version of Gateway and retry? Please let me know if the issue still persists.

https://learn.microsoft.com/en-us/data-integration/gateway/service-gateway-monthly-updates

Thank you

Re: Fabric Dataflow Gen2 Lakehouse: Switch from "new" to "existing" table automatically

Alven — Mon, 06 May 2024 07:47:39 GMT

Hi Anonymous,

as mentioned by Anonymous, it can be helpful to send us the screenshot of the error with the session id but, from what I see from your initial screenshot, this seems a known bug for which we are working on a fix.

Best regards,

Alessandro