Dataflow stopped saving historical data

[sangeetaraman]

Hello,

I had set up a Dataflow (source being Azure Databricks) recently and set it up for refresh everyday. The source displays only current day's data, meaning Azure gives only today's data. In order to save the historical data, I had set up a dataflow which was also working fine by "appending" today's data to past data. I suddenly notice since two days that this has stopped and now each day, the data is "replaced" by current day's data from Azure. I need the daily new data to be appended into the dataflow instead of getting replaced as my visuals need to be showing history as well. Please guide me where I'm going wrong. Sharing the refresh settings set up currently:

Thanks

Sangeeta

[sangeetaraman]

Hi everyone,

An update to my query above- My dataflow is not refreshing incrementally-each day after the refesh the data gets replaced by today's data. But in the desktop, the incremental refresh is working fine. This is strange, can anyone help me on the difference between incremental refresh in the Power BI service and Power BI desktop? What will the consequences of this be after I publish my report online?

Any help is appreciated.

Rgds

Sangeeta

[sangeetaraman]

PS: I have this dataflow set up on a premium workspace

[v-xiaosun-msft]

Hi @sangeetaraman ,

Are the settings now the same as when it worked before? Have any changes been made?

When you need Historical Data Preservation using Power BI Dataflow, Create two entities, one for storing transactional data and another for storing historical data.

Entity for transactional data:

Always stores data for the current year. Once the entity is created, schedule it daily as needed, so as to initiate the incremental refresh.

Entity for historical data:

Only stores the historical data for the previous year and older. This entity can be scheduled if needed or can be triggered manually once a year as needed.

You can reference the following document which may be helpful to you.

Historical Data Preservation using Power BI Dataflow - Visual BI Solutions

Best Regards,
Community Support Team _ xiaosun

If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.

[sangeetaraman]

Hello @v-xiaosun-msft ,

Thanks for your response. There was a change introduced in the source- Two new columns were inserted into the same query. I also see another error which was not the case earlier, "Unable to confirm if M Query can be folded"-screenshot below. Can you please guide me how to rectify this?

Thanks

Sangeeta