NEWS::UPDATE::LAKEHOUSE PIPELINE SYNC ISSUE

[Element115]

ISSUE:

After a Copy Data pipeline activity writes data to a lakehouse, for example, creating a new table for the first time or just appending to an existing table, any subsequent activities in the same pipeline or another pipeline depending on this table or data, needs to wait until the write operation is actually committed/finished. 

Currently, there is no simple and elegant way to know when the data write is done and the data is available, so the user needs to add a Wait activity to the pipeline and guess how many minutes to wait.  This in turn will cause the pipeline execution to pause longer than necessary and lengthen its runtime. 

NEWS:

As per the support team I am working with, Microsoft announced they are rolling out a fix with a Q2 ETA.  

The fix will consist in giving users the ability to issue a REST call against the lakehouse to check whether the data write has completed.  Upon an affirmative response, the user logic is responsible for resuming the pipeline execution.  

If you do not like the idea of having to use a REST API call, and rather wished there would be a pipeline activity that would be easily configurable with the lakehouse name from a drop down list and that would take care automatically of polling the lakehouse and resume pipeline execution upon receipt of a green light without you the user having to mess around configuring a REST API call, or having to drop down to a Notebook and write Python code and messing with authorization tokens, connection strings and the such, THEN go to the Ideas page 

https://aka.ms/FabricIdeas

and make your wish heard.

Obviously, the more users request the same feature, the higher the chance of getting it. So not only upvote the idea if it already exists, but create a new one that is similar.  If the Ideas board is flooded with hundreds of similar ideas, the idea gets more noticed.

[v-csrikanth · Accepted solution]

Hi @Element115 

Thank you for bringing up this issue regarding data availability delays in Lakehouse after a Copy Data pipeline activity. We understand the challenges of relying on an estimated Wait Activity duration, which can lead to longer pipeline execution times.

As mentioned, Microsoft Support team is working on a new REST API feature that will allow users to check when the data write operation is fully committed. This enhancement will improve synchronization and pipeline efficiency.

We appreciate your feedback, and if you would like a native pipeline activity that automatically handles this synchronization (without REST API configuration), please support the idea on Fabric Ideas.

The more votes an idea gets, the higher the priority Microsoft assigns to it! 🚀

Let us know if you need further assistance. We are here to help!

Please accept your response and so that other community will be helpful if they faces similar issue.

Best Regards,
Cheri Srikanth.

[Element115]

@v-csrikanth 

Here is a workaround originally shared by Mark Pryce-Maher, the 

Microsoft PM responsible for the synchronisation between lakehouse and SQL endpoint. As he says:  "the following solution is unofficial, unsupported, undocumented, and frankly unwise!"  

However, I would not call this a solution since it is unofficial, unsupported, undocumented, and unwise.  Anything that is unofficial, unsupported, undocumented, is precisely unwise because it could change at any time without any notice to the user, who will be much bewildered when stuff suddenly breaks for no apparent reason.

Mark's original Python code:  Workaround for delays in the automatically generated schema in the SQL analytics endpoint of the Lak...

An improved version of this code by Andre Fomin:  Fix SQL Analytics Endpoint Sync Issues in Microsoft Fabric – Data Not Showing? Here's the Solution! ...

For now, this Python script seems to work and the problem I had due to the sync issue has not manifested itself... yet.