Forum Discussion
QUESTION::PIPELINE::COPY DATA::DUPLICATE DATA COPY
Here is a workaround originally shared by Mark Pryce-Maher, the
Microsoft PM responsible for the synchronisation between lakehouse and SQL endpoint. As he says: "the following solution is unofficial, unsupported, undocumented, and frankly unwise!"
However, I would not call this a solution since it is unofficial, unsupported, undocumented, and unwise. Anything that is unofficial, unsupported, undocumented, is precisely unwise because it could change at any time without any notice to the user, who will be much bewildered when stuff suddenly breaks for no apparent reason.
Mark's original Python code: Workaround for delays in the automatically generated schema in the SQL analytics endpoint of the Lakehouse in Fabric | by Mark Pryce-Maher | Medium
An improved version of this code by Andre Fomin: Fix SQL Analytics Endpoint Sync Issues in Microsoft Fabric – Data Not Showing? Here's the Solution! — obvience
For now, this Python script seems to work and the problem I had due to the sync issue has not manifested itself... yet.
I’ve gotten the same issue using Copy activity when copying data from a SQL Server database into a Lakehouse as Parquet files.
Intermittently, the copy completes successfully, but the resulting data contains large numbers of duplicate records.
For example, in my case:
The source table normally contains 112,668,879 records (this is the expected unique count).
In one run, the Lakehouse ended up with 177,468,879 records, all the additional records were duplicates, 64,800,000 duplicate records exactly. Given the data is loaded in batches, this round number indicates something went wrong there.
There were no changes to the source data or pipeline configuration between runs.
Has anyone else encountered this behavior, or identified why Copy activity might occasionally produce duplicate records when writing to a Lakehouse?