Forum Discussion

alozovoy's avatar
alozovoy
Icon for Advocate II rankAdvocate II
2 years ago

Duplicated Rows In Tables Built By Notebook

I was seeing incorrect numbers after joining two tables together. I have isolated the issue as shown in the following example:   I take the existing delta table named Rainbow_Accounting_Adjustments...
data engineering
frithjof_v's avatar
frithjof_v
Icon for Community Champion rankCommunity Champion
2 years ago

I was able to reproduce a similar issue.
I created a table (TestData) in a Lakehouse by using a dataflow gen2.

Afterwards, I used the same dataflow gen2 to update the data in this table (TestData).
I also created another dataflow gen2 which I also used to write som data to the same table (TestData).
I also switched between append and replace mode.

The last few refreshes have been only replace mode.

(Switching between append and replace, and having two dataflow gen2 which are updating the same table, may be an edge case ğŸ˜Š It's probably not something I am going to be doing a lot)

 

If I query my table now, it looks like this from the various interfaces:

SQL Analytics Endpoint:

 

 

Lakehouse explorer:

 

 

 

Notebook: 

(Here, surprisingly, some extra rows are present. These rows were inserted in a previous write to the table, and should not be included in the current version of the table.)

 

 

alozovoy's avatar
alozovoy
Icon for Advocate II rankAdvocate II
2 years ago

Yes, looks like you were able to reproduce it!

 

I am in contact with Microsoft support, so this is very helpful to have a second example from a different user!

  • v-cboorla-msft's avatar
    v-cboorla-msft
    Icon for Microsoft Employee rankMicrosoft Employee
    2 years ago

    Hi alozovoy 

     

    Thanks for using Microsoft Fabric Community.

    Apologies for the delay in response.

    Could you please provide the support ticket number as it would help us to track for more information.

    Just checking back to see if you have a resolution yet. In case if you have any resolution please do share that same with the community as it can be helpful to others.
    Otherwise, will respond back with the more details and we will try to help.


    Thanks.

    • alozovoy's avatar
      alozovoy
      Icon for Advocate II rankAdvocate II
      2 years ago

      The support ticket is #2401240010003437. It has not yet been resolved.

      • MysticSapphire's avatar
        MysticSapphire
        Frequent Visitor
        2 years ago

        I'm experiencing a similar issue:
        My dataflow Gen2 is storing data in lakehouse. Once in a while the 'replace' table setting in the dataflow Gen2 doesn't seem to work and it results in having the same data copied twice. It only seems to be affecting smaller tables.