Forum Discussion
QUESTION::PIPELINE::COPY DATA::DUPLICATE DATA COPY
Here is a workaround originally shared by Mark Pryce-Maher, the
Microsoft PM responsible for the synchronisation between lakehouse and SQL endpoint. As he says: "the following solution is unofficial, unsupported, undocumented, and frankly unwise!"
However, I would not call this a solution since it is unofficial, unsupported, undocumented, and unwise. Anything that is unofficial, unsupported, undocumented, is precisely unwise because it could change at any time without any notice to the user, who will be much bewildered when stuff suddenly breaks for no apparent reason.
Mark's original Python code: Workaround for delays in the automatically generated schema in the SQL analytics endpoint of the Lakehouse in Fabric | by Mark Pryce-Maher | Medium
An improved version of this code by Andre Fomin: Fix SQL Analytics Endpoint Sync Issues in Microsoft Fabric – Data Not Showing? Here's the Solution! — obvience
For now, this Python script seems to work and the problem I had due to the sync issue has not manifested itself... yet.
Was there ever a satisfactory solution to this problem with Microsoft? I have the same scenario, except my entire data set is ocassionally writing 2x. I run a pre-script to delete all data from the destination, and then quite simply load everything from a lakehouse view into a warehouse table. Sometimes the loads are fine, sometimes they are completely doubled. (A CreatedOn field shows that for each duplicated record, there is usually a couple of seconds difference between the load time).
Define 'satisfactory' because way back when I posted the workaround put forth by the Microsoft PM himself. This year, Microsoft finally released a lakehouse REST API in GA. Haven't used it yet. My guess is you drop a web activity in the pipeline and do a REST call to the lakehouse to force a sync, same as is done in the Python notebook.