Forum Discussion
DataFlow Gen2 runs successfully but data is missing in Data Warehouse
Hi ilseeb, I have noticed that sometimes 160 seconds will not be enough to update the SQL-endpoint of a lakehouse. I found a blog posts that contains a Python script to refresh the SQL-endpoint programmatically and wait for the lakehouse to be refreshed before moving on. You can find the blog post here: https://www.obvience.com/blog/fix-sql-analytics-endpoint-sync-issues-in-microsoft-fabric-data-not-showing-heres-the-solution. You can add this in a notebook and run that after copying the data and before appending and transforming.
Hi, are you sure your data in the Lakehouse is being updated before it goes to the warehouse?
In my case, I would check later once I realized data was missing in the warehouse, and data was available in the lakehouse, but what was really happenning is that the lakehosue endpoint was not being refreshed before the data was written in the warehouse. I added a step to refresh the SQL lakehouse endpoint before the warehouse step and the problem was solved.
Hi, the data in the Lakehouse was updated beacuse we implemented a pipeline for a Medallion Architecture: the Lakehouse is the SILVER layer and the Warehouse is the GOLD layer.
The pipeline has been in production for over six months and so far there have been no issues refreshing the Lakehouse endpoint.
If necessary, we can consider the possibility to set a wait task or to force the refresh of the endpoint. But since the problem has not occurred previously, it can be a Fabric issue.