Forum Discussion
Dataflow Gen 2 output differs from Lakehouse table values - Data inconsistency issue
I have a Dataflow Gen 2 that give me a result like this:
Project (str) || Pending Tasks (int) || Overdue Task (int) || dataDate (datetime)
P406 || 59 || 5 || 18-11-2025 12:15:00
P426 || 1 || 0 || 18-11-2025 12:15:00
P418 || 6 || 2 || 18-11-2025 12:15:00
These results are loaded into a Lakehouse, anexing them into a specific table. There is no other step between the obtained result and the writing. The problem is the data I read in that table after the writing:
Project (str) || Pending Tasks (int) || Overdue Task (int) || dataDate (datetime)
P406 || 78 || 23 || 18-11-2025 12:15:00
P426 || 8 || 5 || 18-11-2025 12:15:00
P418 || 6 || 2 || 18-11-2025 12:15:00
There is a discrepance between the Dataflow results and the writing. I even duplicate the code (in the same dataflow) and loaded this duplicate M Query into a Warehouse and obtained another different writing:
Project (str) || Pending Tasks (int) || Overdue Task (int) || dataDate (datetime)
P406 || 2 || 36 || 18-11-2025 12:15:00
P426 || 21 || 10 || 18-11-2025 12:15:00
P418 || 1 || 1 || 18-11-2025 12:15:00
I really don't understand whats happening here, the data was writing perfectly fine 2 or 3 days ago and received no modification.
The complete flow consists in obtain data with a notebook, from an API; save that data into a Lakehouse; retrieve that data from a Dataflow and ETL that (i know this is, maybe, an inefficient way to work); and then use that ETLed data to feed a PBI panel. (This is real data btw)
Is there a known problem that explains this?
Thanks.
Hi FVarelaI,
Thank you tayloramy, fro your insights.
Dataflow Gen2 data destinations and managed settings - Microsoft Fabric | Microsoft Learn
Data Factory limitations overview - Microsoft Fabric | Microsoft Learn
Thank you.
8 Replies
Hi FVarelaI,
Are you connecting to the SQL Endpoint of the lakehouse? It is known that the SQL Endpoint takes time to catch up with the underlaying delta tables.
After your notebook is done writing data, try forcing a metadata sync:
Refresh SQL analytics endpoint Metadata REST API (Generally Available) | Microsoft Fabric Blog | Microsoft FabricIf you found this helpful, consider giving some Kudos. If I answered your question or solved your problem, mark this post as the solution.
Hi FVarelaI,
Thank you tayloramy, fro your insights.
Dataflow Gen2 data destinations and managed settings - Microsoft Fabric | Microsoft Learn
Data Factory limitations overview - Microsoft Fabric | Microsoft Learn
Thank you.
Hi FVarelaI,
Checking in to see if your issue has been resolved. let us know if you still need any assistance.
Thank you.
Hi v-saisrao-msft,
I consider this issue resolved. I’ve been reviewing the outputs over the past few days and haven’t noticed any inconsistencies.
Thank you for your help!
Hi FVarelaI,
Thank you for the update. As the issue has been resolved, please mark the helpful response as the solution so it can assist other community members.
Hi FVarelaI,
Have you had a chance to review the solution we shared earlier? If the issue persists, feel free to reply so we can help further.
Thank you.
Hi v-saisrao-msft !,
I can't use the solution given by tayloramy , because of limitations, partly, of my organization, and partly my own capacities.
Therefore, I found another solution optimizing my dataflow, im not sure enough of what modification solved this problem, but works perfectly fine... for now.
Thanks for your help, it was really useful.Hi FVarelaI,
This issue might be intermittent. If it occurs again, please let us know so we can investigate further.
Thank you!!
