Forum Discussion
Advocate IDataFlow Gen2 runs successfully but data is missing in Data Warehouse
Hi ilseeb, I have noticed that sometimes 160 seconds will not be enough to update the SQL-endpoint of a lakehouse. I found a blog posts that contains a Python script to refresh the SQL-endpoint programmatically and wait for the lakehouse to be refreshed before moving on. You can find the blog post here: https://www.obvience.com/blog/fix-sql-analytics-endpoint-sync-issues-in-microsoft-fabric-data-not-showing-heres-the-solution. You can add this in a notebook and run that after copying the data and before appending and transforming.
Advocate IHello, I had the same issue some days ago.
I have a Dataflow Gen2 that reads many tables from a Lakehouse, makes some transformation and finally write modified data in a Warehouse overwriting the destination tables.
Although execution logs indicate that data was read and written without errors, tables in the Warehouse doesn't contain any new or updated data, as if no overwrite happened. Except for the CALENDAR table, that is generated inside the Dataflow Gen2 using only M code (no source from the Lakehouse).
In the Lakehouse, new and updated records are available. Checking Dataflow steps today, I can see those records.
I would know if there is a official planned solution for 2026 to this problem.