Forum Discussion
Advocate IIDuplicated Rows In Tables Built By Notebook
Skilled SharerSame issue, any solution?
In my case the issue resoled itself when the dataflow re-ran. (In my case at least) it looks like the dataflow failed to "version" the Delta Table in the lakehouse, and for some reason only Notebooks then return "multiple versions" of the table (the new data AND the old data).
Re-running the dataflow might resolve your issue, or if all else fails, delete the table in the lakehouse manually and re-run the dataflow.
MS was not able to provide me with a solution or explaination since by the time they got in touch with me the issue had resolved itself (the next day).
- amaaiia
In Feb 19th I got same issue with another table (check my post), and what I did is what you are saying, delete manually and start again. But as this has happened again, I think it should be a solution by MS side. In my case, I didn't have any issue when overwriting the table through DFg2 as you mentioned you had with your 6 tables, all my tables where succesfully loaded with no errors in DFg2 publishing, so I can't understand why is this happening.
I guess I'm creating new support ticket...
alozovoyRather than manually deleting, you can set up a notebook to run a delete script each time before your dataflow executes. Here's an example I have that runs nightly before my dataflows: