Forum Discussion
Duplicate rows via a staging table
Hi ToddChitt,
Thank you for reaching out to Microsoft Fabric Community Forum.
Verify whether the FULL and INCREMENTAL loads are overlapping or interfering with each other. If the FULL load takes longer than expected, an INCREMENTAL load might begin while the table is still being truncated and reloaded, potentially causing duplicates.
If there is no overlap and the issue persists, I recommend raising a Microsoft Fabric support ticket for further investigation.
How to create a Fabric and Power BI Support ticket - Power BI | Microsoft Learn
Regards,
Vinay Pabbu
I know that in Microsoft SQL Server, NULL <> NULL (Meaning: If you have a value of NULL in BOTH corresponding fields of a JOIN, you will NOT get a match.) I am sort of assuming that Fabric follows that same rule.
But that key column is not null. Anywhere.
I agree with you that the join will not let NULL values pass the criteria. However, it could have ended up in the target with a FULL load, which then potentielly creates duplicates in the upsert logica after a new INCREMENTAL load is initiated.
No. While I do not have access to inspect the metadata of the source table, it is a SQL Server database, with that column being the UNIQUE KEY for the table, I suspect it is also defined as NOT NULL..
Perfect! Then I'm out of options for now