Forum Discussion

alozovoy's avatar
alozovoy
Icon for Advocate II rankAdvocate II
2 years ago

Duplicated Rows In Tables Built By Notebook

I was seeing incorrect numbers after joining two tables together. I have isolated the issue as shown in the following example:   I take the existing delta table named Rainbow_Accounting_Adjustments...
data engineering
amaaiia's avatar
amaaiia
Icon for Skilled Sharer rankSkilled Sharer
2 years ago

Same issue, any solution?

Anonymous's avatar
Anonymous
Not applicable
2 years ago

In my case the issue resoled itself when the dataflow re-ran.  (In my case at least) it looks like the dataflow failed to "version" the Delta Table in the lakehouse, and for some reason only Notebooks then return "multiple versions" of the table (the new data AND the old data).

 

Re-running the dataflow might resolve your issue, or if all else fails, delete the table in the lakehouse manually and re-run the dataflow.

 

MS was not able to provide me with a solution or explaination since by the time they got in touch with me the issue had resolved itself (the next day).

  • amaaiia's avatar
    amaaiia
    Icon for Skilled Sharer rankSkilled Sharer
    2 years ago

    In Feb 19th I got same issue with another table (check my post), and what I did is what you are saying, delete manually and start again. But as this has happened again, I think it should be a solution by MS side. In my case, I didn't have any issue when overwriting the table through DFg2 as you mentioned you had with your 6 tables, all my tables where succesfully loaded with no errors in DFg2 publishing, so I can't understand why is this happening.

     

    I guess I'm creating new support ticket...

    • alozovoy's avatar
      alozovoy
      Icon for Advocate II rankAdvocate II
      2 years ago

      Rather than manually deleting, you can set up a notebook to run a delete script each time before your dataflow executes. Here's an example I have that runs nightly before my dataflows:

       

       

      • frithjof_v's avatar
        frithjof_v
        Icon for Community Champion rankCommunity Champion
        2 years ago

        I'm curious, will this work if you are using this table in a relationship in a Semantic model?

         

        I would assume any relationships would get broken? (Or that the table cannot be dropped if there is a relationship)

         

        Or if there is a OneLake shortcut pointing to this table, I would assume that any shortcut would get broken when dropping the table?

         

        Anyone knows / have tried?