Forum Discussion
Duplicate rows via a staging table
Hi ToddChitt,
Thank you for reaching out to Microsoft Fabric Community Forum.
Verify whether the FULL and INCREMENTAL loads are overlapping or interfering with each other. If the FULL load takes longer than expected, an INCREMENTAL load might begin while the table is still being truncated and reloaded, potentially causing duplicates.
If there is no overlap and the issue persists, I recommend raising a Microsoft Fabric support ticket for further investigation.
How to create a Fabric and Power BI Support ticket - Power BI | Microsoft Learn
Regards,
Vinay Pabbu
Hello Anonymous and thank you for the support link. I will keep it in mind.
I can verify that the FULL and DELTA loads are NOT overlapping. FULL happens on Sundays, DELTA happens Monday through Saturday. Times are roughly 1 hour and 20 minutes, respectively, with multiple tables involved in each load.
I think the thing to do is set up an additional set of Staging Table in the Lakehouse, Destination table in the Warehouse, 2 additional rows in the Control Table for FULL and DELTA loads, and a totally new set of pipelines to handle just this one-off set of tables. That way, if I get duplicates, I can halt any additional loading until Microsoft Support can examine the Run ID of the pipelines that generated the duplicates.
If I get a resolution, I will try to post back here, but no promises.
Hi ToddChitt,
Since we haven't heard back from you, we wanted to follow up regarding your ticket.
Could you please provide an update on the status of your ticket ? it will be helpful for other members of the community who have similar problems as yours to solve it faster.
Regards,
Vinay Pabbu