Forum Discussion
Duplicated Rows In Tables Built By Notebook
The support ticket is #2401240010003437. It has not yet been resolved.
I'm experiencing a similar issue:
My dataflow Gen2 is storing data in lakehouse. Once in a while the 'replace' table setting in the dataflow Gen2 doesn't seem to work and it results in having the same data copied twice. It only seems to be affecting smaller tables.
I believe to have identified the root cause: In some cases my DataflowGen2 would fail with one table. For 6 of my tables the replace load was completed without issues in the same flow. However, at the 'Write' step for the last table it would fail causing the full dataflow Gen 2 to fail (with 6 tables completely loaded and 1 failed).
In the above scenario those 6 tables would somehow have persisted duplicates when I queried it via Lakehouse. I could only get rid of them by dropping the table and reloading the dataflow. As long as my dataflow gen 2 doesn't fail, I don't get the duplicate issues it seems.
However as alozovoy pointed out, running a notebook dropping all the tables prior to the replace seems the best robust approach for now untill Microsoft fixes this quirck