Forum Discussion
Duplicated Rows In Tables Built By Notebook
Yes, looks like you were able to reproduce it!
I am in contact with Microsoft support, so this is very helpful to have a second example from a different user!
Hi alozovoy
Thanks for using Microsoft Fabric Community.
Apologies for the delay in response.
Could you please provide the support ticket number as it would help us to track for more information.
Just checking back to see if you have a resolution yet. In case if you have any resolution please do share that same with the community as it can be helpful to others.
Otherwise, will respond back with the more details and we will try to help.
Thanks.
The support ticket is #2401240010003437. It has not yet been resolved.
I'm experiencing a similar issue:
My dataflow Gen2 is storing data in lakehouse. Once in a while the 'replace' table setting in the dataflow Gen2 doesn't seem to work and it results in having the same data copied twice. It only seems to be affecting smaller tables.I believe to have identified the root cause: In some cases my DataflowGen2 would fail with one table. For 6 of my tables the replace load was completed without issues in the same flow. However, at the 'Write' step for the last table it would fail causing the full dataflow Gen 2 to fail (with 6 tables completely loaded and 1 failed).
In the above scenario those 6 tables would somehow have persisted duplicates when I queried it via Lakehouse. I could only get rid of them by dropping the table and reloading the dataflow. As long as my dataflow gen 2 doesn't fail, I don't get the duplicate issues it seems.
However as alozovoy pointed out, running a notebook dropping all the tables prior to the replace seems the best robust approach for now untill Microsoft fixes this quirck
