Forum Discussion
Dataflow Gen2 not updating Lakehouse
Hello,
My dataflow runs correctly with no errors. I have a conditional column in a table I save to my lakehouse. The conditional adds today's date in case the date column is empty. The date is then used to filter the table in other reports.
What keeps happening is the date is not refreshed always. The dataflow is scheduled to refresh daily (through a pipeline). The refresh is always successful. But today for example, the date in the Lakehouse remained 17/11/2024 instead of 18/11/2024.
Any idea why this is happening? I use the table saved in the lakehouse in multiple other reports, and this is screwing up the results in those other reports.
on th 17/11, the dataflow was refreshed at 07:19 AM, successfully via a pipeline. the lakehouse was updated accordingly.
This morning, the dataflow was refreshed at 07:21 AM, successfully via a pipeline. the lakehouse was not updated accordingly.
I refreshed the dataflow manully at 10:42 AM. the lakehouse was updated accordingly.
8 Replies
Hi ina_dab
Thank you very much SudhavaniKolla3 for your prompt reply.
Make sure you have sufficient permissions. There may be a synchronization delay between the dataflow and the lakehouse, resulting in updates in the dataflow not being reflected in the lakehouse in a timely manner.
Here are some suggestions:
Ensure that the output of dataflow is set correctly to write data to lakehouse in a timely manner.
If possible, consider increasing the refresh frequency of the dataflow to minimize the delay between data updates.
Examine the logic in the dataflow to ensure that there are no redundant steps or complex calculations causing delays.
Looking forward to you sharing more.
Regards,
Nono Chen
If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.
Hello, I am admin of my fabric environment so there is no permissions issue.
The issue really is that this specific table from the dataflow is not writing into the lakehouse. Let's call this Table A. Table A is a transformation of two other tables, one of them being Table B. This date column is in table B, it doesn't even get transformed in Table A. Table B is showing the correct date in my lakehouse, just Table A is not.
I have deleted the lakehouse table before and made sure the mapping is again correct, but I still don't get it to write correctly.
frithjof_vCommunity Champion
How are you checking what data you have in the Lakehouse?
Are you checking it in the SQL Analytics Endpoint (there may be a delay between the actual Lakehouse and the SQL Analytics Endpoint)?
How is your Power BI semantic model connected to the Lakehouse? Via Direct Lake or Import mode?
- SudhavaniKolla3
Helper II
the default date is may be creating UTC time which is some hours difference to IST or other timings.
try to convert that time to your local time and see
If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.Time is currently set to my local timezone. Nonetheless, that can't be the issue. The pipeline is activated daily at the same time. the refresh from yesterday was 2 minutes earlier than the one today, yet that one updated the lakehouse, the one from today did not.
Also, if I am just to open the dataflow and look at the table there, the date is updated but the lakehouse is not.
Hi ina_dab
Thank you for your feedback!
I would suggest that you can create a support ticket and a professional engineer from Microsoft will contact you to resolve the issue for you.
It would be great if you could continue to share the issue once you know the root cause or solution to help others who are experiencing similar issues.
Power BI support link: https://powerbi.microsoft.com/en-us/support/
For more information on how to create a support ticket, please see How to Create a Support Ticket in Power BI - Microsoft Fabric Community
Thank you for your understanding.
Regards,
Nono Chen
I have exactly the same problem.
Nothing works. I did not even change the Dataflows.
