Forum Discussion
Duplicate rows via a staging table
Hi ToddChitt,
Thank you for reaching out to Microsoft Fabric Community Forum.
Verify whether the FULL and INCREMENTAL loads are overlapping or interfering with each other. If the FULL load takes longer than expected, an INCREMENTAL load might begin while the table is still being truncated and reloaded, potentially causing duplicates.
If there is no overlap and the issue persists, I recommend raising a Microsoft Fabric support ticket for further investigation.
How to create a Fabric and Power BI Support ticket - Power BI | Microsoft Learn
Regards,
Vinay Pabbu
I have the exact same issue occuring in my Fabric, in this case I am using a Fabric Warehouse so there is no key constraint present in the source/destination. The Stored Procedure does largely the same thing as yours but also does a ROW_NUMBER() operation that increments when a new row is added, which happens only when the key is missing via WHERE NOT EXISTS. When I group by this Key I get 2x of everything!
The conclusion must surely be that the Stored Procedure somehow ran twice within a short time-frame and then inserted both sets of data even though they were duplicate. This is evident by the fact that this merge proc is quite large on that initial load and takes over 10 minutes, yet on the table the duplicated inserts are within 5 minutes of eachother.
Notably in my case, this only happens on the initial load of the biggest table (Over 100 Million rows) in the Data Warehouse, after that point the process appears to behave itself and only inserts one set of data.
Upon further inspection I think I have found the root cause of my issue at least, I think by an unhappy coincidence my full-load did collide with a scheduled refresh. I would double/triple check that your FULL LOAD isn't also running at the same time as your DELTAs, or another scheduled process is being kicked off at the same time somewhere else.
Will do a full reload tonight and just double check that this is what caused it on my end.