Forum Discussion
DataFlow Gen2 runs successfully but data is missing in Data Warehouse
Hi ilseeb, I have noticed that sometimes 160 seconds will not be enough to update the SQL-endpoint of a lakehouse. I found a blog posts that contains a Python script to refresh the SQL-endpoint programmatically and wait for the lakehouse to be refreshed before moving on. You can find the blog post here: https://www.obvience.com/blog/fix-sql-analytics-endpoint-sync-issues-in-microsoft-fabric-data-not-showing-heres-the-solution. You can add this in a notebook and run that after copying the data and before appending and transforming.
Hi ilseeb, I have noticed that sometimes 160 seconds will not be enough to update the SQL-endpoint of a lakehouse. I found a blog posts that contains a Python script to refresh the SQL-endpoint programmatically and wait for the lakehouse to be refreshed before moving on. You can find the blog post here: https://www.obvience.com/blog/fix-sql-analytics-endpoint-sync-issues-in-microsoft-fabric-data-not-showing-heres-the-solution. You can add this in a notebook and run that after copying the data and before appending and transforming.
Thanks a lot for this link! I'll try it out
Hi ilseeb
Thank you very much FabianSchut for your prompt reply.
Can you tell me if your problem is solved? If yes, please accept it as solution.
Regards,
Nono Chen
If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.
ilseeb FabianSchut Anonymous
Unfortunately, this is a gamble. Being an undocumented and unsupported solution by Microsoft, as soon as one morning some Microsoft tech bro parachutes into his cubicle all fresh and perky, full of unwarranted optimism for the day ahead fueled by too much caffeine, and on a whim decides to change some code in the API, this 'so-called' solution will come crashing down while you run it in production.
Who wants to gamble with production workloads? Not me.
But since Microsoft has been working on this and they say they will release something in Q2, like a REST API endpoint to query in order to refresh the lakehouse metadata (I guess they are just officializing this undocumented solution), why not send them a message, loud and clear, by voting for the idea found at the link below and that will give us a new pipeline activity that's easy to set up from within a pipeline, set and forget, and will do the job without having to mess around with a Python Notebook? (I have in mind here those users that expect user-friendliness, and not those who are coding veterans.)
Here is the idea link: LAKEHOUSE I/O WRITE DELAY MITIGATION - Microsoft Fabric Community
