topic Re: Dataflow Gen2 not updating Lakehouse in Dataflow

Dataflow Gen2 not updating Lakehouse

ina_dab — Mon, 18 Nov 2024 10:02:30 GMT

Hello,

My dataflow runs correctly with no errors. I have a conditional column in a table I save to my lakehouse. The conditional adds today's date in case the date column is empty. The date is then used to filter the table in other reports.

What keeps happening is the date is not refreshed always. The dataflow is scheduled to refresh daily (through a pipeline). The refresh is always successful. But today for example, the date in the Lakehouse remained 17/11/2024 instead of 18/11/2024.

Any idea why this is happening? I use the table saved in the lakehouse in multiple other reports, and this is screwing up the results in those other reports.

on th 17/11, the dataflow was refreshed at 07:19 AM, successfully via a pipeline. the lakehouse was updated accordingly.

This morning, the dataflow was refreshed at 07:21 AM, successfully via a pipeline. the lakehouse was not updated accordingly.

I refreshed the dataflow manully at 10:42 AM. the lakehouse was updated accordingly.

Re: Dataflow Gen2 not updating Lakehouse

SudhavaniKolla3 — Mon, 18 Nov 2024 12:09:23 GMT

the default date is may be creating UTC time which is some hours difference to IST or other timings.

try to convert that time to your local time and see
If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.

Re: Dataflow Gen2 not updating Lakehouse

ina_dab — Mon, 18 Nov 2024 12:58:38 GMT

Time is currently set to my local timezone. Nonetheless, that can't be the issue. The pipeline is activated daily at the same time. the refresh from yesterday was 2 minutes earlier than the one today, yet that one updated the lakehouse, the one from today did not.

Also, if I am just to open the dataflow and look at the table there, the date is updated but the lakehouse is not.

Re: Dataflow Gen2 not updating Lakehouse

Anonymous — Tue, 19 Nov 2024 02:14:15 GMT

Hi ina_dab

Thank you very much SudhavaniKolla3 for your prompt reply.

Make sure you have sufficient permissions. There may be a synchronization delay between the dataflow and the lakehouse, resulting in updates in the dataflow not being reflected in the lakehouse in a timely manner.

Here are some suggestions:

Ensure that the output of dataflow is set correctly to write data to lakehouse in a timely manner.

If possible, consider increasing the refresh frequency of the dataflow to minimize the delay between data updates.

Examine the logic in the dataflow to ensure that there are no redundant steps or complex calculations causing delays.

Looking forward to you sharing more.

Regards,

Nono Chen

If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.

Re: Dataflow Gen2 not updating Lakehouse

ina_dab — Wed, 20 Nov 2024 08:31:18 GMT

Hello, I am admin of my fabric environment so there is no permissions issue.

The issue really is that this specific table from the dataflow is not writing into the lakehouse. Let's call this Table A. Table A is a transformation of two other tables, one of them being Table B. This date column is in table B, it doesn't even get transformed in Table A. Table B is showing the correct date in my lakehouse, just Table A is not.
I have deleted the lakehouse table before and made sure the mapping is again correct, but I still don't get it to write correctly.

Re: Dataflow Gen2 not updating Lakehouse

Anonymous — Wed, 20 Nov 2024 09:05:39 GMT

Hi ina_dab

Thank you for your feedback!

I would suggest that you can create a support ticket and a professional engineer from Microsoft will contact you to resolve the issue for you.

It would be great if you could continue to share the issue once you know the root cause or solution to help others who are experiencing similar issues.

Power BI support link: https://powerbi.microsoft.com/en-us/support/

For more information on how to create a support ticket, please see How to Create a Support Ticket in Power BI - Microsoft Fabric Community

Thank you for your understanding.

Regards,

Nono Chen

Re: Dataflow Gen2 not updating Lakehouse

frithjof_v — Tue, 26 Nov 2024 20:27:01 GMT

How are you checking what data you have in the Lakehouse?

Are you checking it in the SQL Analytics Endpoint (there may be a delay between the actual Lakehouse and the SQL Analytics Endpoint)?

How is your Power BI semantic model connected to the Lakehouse? Via Direct Lake or Import mode?

Re: Dataflow Gen2 not updating Lakehouse

DHC-Controlling — Mon, 02 Dec 2024 12:19:36 GMT

I have exactly the same problem.
Nothing works. I did not even change the Dataflows.

Re: Dataflow Gen2 not updating Lakehouse

MiSchroe — Mon, 31 Mar 2025 15:30:23 GMT

Kudos to frithjof_v ! I have a similar problem where a column shouldn't contain NULL values after the second run of the dataflow. But the column still contains NULL values when I query it through the SQL endpoint but it is filled correctly when I query the table with a Spark SQL notebook.