Forum Discussion
QUESTION::PIPELINE::COPY DATA::DUPLICATE DATA COPY
Here is a workaround originally shared by Mark Pryce-Maher, the
Microsoft PM responsible for the synchronisation between lakehouse and SQL endpoint. As he says: "the following solution is unofficial, unsupported, undocumented, and frankly unwise!"
However, I would not call this a solution since it is unofficial, unsupported, undocumented, and unwise. Anything that is unofficial, unsupported, undocumented, is precisely unwise because it could change at any time without any notice to the user, who will be much bewildered when stuff suddenly breaks for no apparent reason.
Mark's original Python code: Workaround for delays in the automatically generated schema in the SQL analytics endpoint of the Lakehouse in Fabric | by Mark Pryce-Maher | Medium
An improved version of this code by Andre Fomin: Fix SQL Analytics Endpoint Sync Issues in Microsoft Fabric – Data Not Showing? Here's the Solution! — obvience
For now, this Python script seems to work and the problem I had due to the sync issue has not manifested itself... yet.
Hi Element115,
Thank you for your follow-up. I apologise for any confusion caused by the initial response.
Based on my understanding, your configuration appears to be correct as per the details shared. The issue seems to arise from unexpected behaviour within the Copy Data activity, rather than any user-related error.
Considering the irregular duplication such as 8 rows being copied instead of the expected 4, without any consistent pattern as shown in your recent screenshot it appears to be a platform-related behaviour. As you have already raised a support ticket, we kindly request you to continue collaborating closely with the support team. Please share detailed logs and observations with them to aid in a comprehensive investigation.
We would appreciate it if you could keep us informed about the resolution of the support ticket, as it may prove beneficial for the wider Fabric community.
If you find our response helpful, we request you to mark it as the accepted solution and consider giving kudos. This will help other community members who may encounter similar queries.
Thank you.