Forum Discussion
DataFlow Gen2 runs successfully but data is missing in Data Warehouse
Hi ilseeb, I have noticed that sometimes 160 seconds will not be enough to update the SQL-endpoint of a lakehouse. I found a blog posts that contains a Python script to refresh the SQL-endpoint programmatically and wait for the lakehouse to be refreshed before moving on. You can find the blog post here: https://www.obvience.com/blog/fix-sql-analytics-endpoint-sync-issues-in-microsoft-fabric-data-not-showing-heres-the-solution. You can add this in a notebook and run that after copying the data and before appending and transforming.
Hello, I had the same issue some days ago.
I have a Dataflow Gen2 that reads many tables from a Lakehouse, makes some transformation and finally write modified data in a Warehouse overwriting the destination tables.
Although execution logs indicate that data was read and written without errors, tables in the Warehouse doesn't contain any new or updated data, as if no overwrite happened. Except for the CALENDAR table, that is generated inside the Dataflow Gen2 using only M code (no source from the Lakehouse).
In the Lakehouse, new and updated records are available. Checking Dataflow steps today, I can see those records.
I would know if there is a official planned solution for 2026 to this problem.
Hi, are you sure your data in the Lakehouse is being updated before it goes to the warehouse?
In my case, I would check later once I realized data was missing in the warehouse, and data was available in the lakehouse, but what was really happenning is that the lakehosue endpoint was not being refreshed before the data was written in the warehouse. I added a step to refresh the SQL lakehouse endpoint before the warehouse step and the problem was solved.Hi, the data in the Lakehouse was updated beacuse we implemented a pipeline for a Medallion Architecture: the Lakehouse is the SILVER layer and the Warehouse is the GOLD layer.
The pipeline has been in production for over six months and so far there have been no issues refreshing the Lakehouse endpoint.
If necessary, we can consider the possibility to set a wait task or to force the refresh of the endpoint. But since the problem has not occurred previously, it can be a Fabric issue.
