Forum Discussion

Miszel8's avatar
Miszel8
Frequent Visitor
2 years ago

Fabric Delta Tables - issue with duplicates

Hello,

 

I've been using Databricks witht delta like for a while. I have all the delta tables (parquet files + delta metada catalogue) stored on ADLS2.

 

I wanted to test Farbic, so I've exported the data to Fabric Lakehouse (cannot use shortcut due to VNET policy).

And here is my issue, after converting the data from files, to tables, each of my tables contains tons of duplicated rows.

Here is the example:

Should be only 3 rows of the data in this dimension instead of 27.

In the meantime the same files are read correctly by Databricks.

Do you know how to solve this problem?

 

BTW - it seems there is no community dedicated to fabric, any ideas where should I look for help?

Need Help

2 Replies

  • GilbertQ's avatar
    GilbertQ
    Icon for Super User rankSuper User
    2 years ago

    Hi Miszel8 

     

    I would suggest looking at the way you are importing the data into the lake house tables. It looks to me that you are using an append instead of an overwrite  And that is why, for each time the file is being imported, it is appending to the existing data.

    • Miszel8's avatar
      Miszel8
      Frequent Visitor
      2 years ago

      Thank you for the answer, however the tables were loaded only once - no additional appending/overwriting took place.