Forum Discussion
Memorable MemberNEWS::UPDATE::LAKEHOUSE PIPELINE SYNC ISSUE
Hi Element115
Thank you for bringing up this issue regarding data availability delays in Lakehouse after a Copy Data pipeline activity. We understand the challenges of relying on an estimated Wait Activity duration, which can lead to longer pipeline execution times.
As mentioned, Microsoft Support team is working on a new REST API feature that will allow users to check when the data write operation is fully committed. This enhancement will improve synchronization and pipeline efficiency.We appreciate your feedback, and if you would like a native pipeline activity that automatically handles this synchronization (without REST API configuration), please support the idea on Fabric Ideas.
The more votes an idea gets, the higher the priority Microsoft assigns to it! 🚀
Let us know if you need further assistance. We are here to help!
Please accept your response and so that other community will be helpful if they faces similar issue.
Best Regards,
Cheri Srikanth.
Community SupportHi Element115
Thank you for bringing up this issue regarding data availability delays in Lakehouse after a Copy Data pipeline activity. We understand the challenges of relying on an estimated Wait Activity duration, which can lead to longer pipeline execution times.
As mentioned, Microsoft Support team is working on a new REST API feature that will allow users to check when the data write operation is fully committed. This enhancement will improve synchronization and pipeline efficiency.
We appreciate your feedback, and if you would like a native pipeline activity that automatically handles this synchronization (without REST API configuration), please support the idea on Fabric Ideas.
The more votes an idea gets, the higher the priority Microsoft assigns to it! 🚀
Let us know if you need further assistance. We are here to help!
Please accept your response and so that other community will be helpful if they faces similar issue.
Best Regards,
Cheri Srikanth.
- Element115
Here is a workaround originally shared by Mark Pryce-Maher, the
Microsoft PM responsible for the synchronisation between lakehouse and SQL endpoint. As he says: "the following solution is unofficial, unsupported, undocumented, and frankly unwise!"
However, I would not call this a solution since it is unofficial, unsupported, undocumented, and unwise. Anything that is unofficial, unsupported, undocumented, is precisely unwise because it could change at any time without any notice to the user, who will be much bewildered when stuff suddenly breaks for no apparent reason.
Mark's original Python code: Workaround for delays in the automatically generated schema in the SQL analytics endpoint of the Lakehouse in Fabric | by Mark Pryce-Maher | Medium
An improved version of this code by Andre Fomin: Fix SQL Analytics Endpoint Sync Issues in Microsoft Fabric – Data Not Showing? Here's the Solution! — obvience
For now, this Python script seems to work and the problem I had due to the sync issue has not manifested itself... yet.
