Incremental Refresh - data being removed from source

[Danfountain]

Hi all,

QQ.  With incremental refresh - if you had a source that was only keeping 2 months data, could incremental refresh be used to keep 12 months data within the report?

Many thanks

Dan

[v-jianpeng-msft]

Hi, @Danfountain 

Incremental refresh based on the policy you set, and here are some detailed explanations:
The following diagram shows the entire workflow for incremental refresh:

The first refresh loads new and updated data from the refresh period, as well as historical data for the entire storage period. 

1.Archive data:

In this example, Power BI deletes data from 5 years prior to today (11/17/2022). Load and save data for 1/1/2017-11/17/2022. If your data is in this range, then they are still valid.

2.Incremental data

In this example, Power BI Service refreshes the data for the last three days today (11/20/2022) 11/18/2022-11/20/2022. If you delete this part of the data, the changes will be reflected in your semantic model. 

Here are some documents that provide insight into incremental refresh:

Configure incremental refresh and real-time data for Power BI semantic models - Power BI | Microsoft...

Incremental refresh for semantic models and real-time data in Power BI - Power BI | Microsoft Learn

A deep dive into Power BI incremental refresh feature | by Jonathan Law | Towards Data Science

Aggregated Intelligence: Understanding PowerBi Incremental Refresh with Detect Changes

Handling of Deleted data in incremental refresh - Microsoft Fabric Community

Solved: How to clear or empty out a Power BI dataset that ... - Microsoft Fabric Community

How to Get Your Question Answered Quickly

If it does not help, please provide more details with your desired output and pbix file without privacy information (or some sample data) .

Best Regards

Jianpeng Li

If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.

[Danfountain]

I have now found another question.

So imagine i have done the above.  The data refreshes nightly and the "deleted" data is preserved in powerBI.

Now lets just say a few months pass and its running.  I then want to make some changes.  As soon as I try to change the model its no longer going to be able to get that data and therefore I will lose it?  If so this might not be the option that i thought it was.

[v-jianpeng-msft · Accepted solution]

Hi, @Danfountain 

If you want to change a semantic model that has been configured for incremental refresh, such as a full refresh of your semantic model, you will lose the data in the Archive data. You'll need to re-upload your PBIX file in Power BI Desktop. You can refer to the case link below:

Solved: How to clear or empty out a Power BI dataset that ... - Microsoft Fabric Community

Incremental refresh for Power BI refreshes are based on the policies that you set. In incremental refresh, when you set an incremental refresh policy, Power BI creates and manages partitions of your data. Only the most recent data that has changed needs to be refreshed. If you have successfully incrementally refreshed several times. If you suddenly have only nearly two months of data in your data source one day, the data in the Archive data compartment will still be able to use that data in Power BI, but it will be phased out according to the policies you set.

While incremental refresh is a powerful feature, it's important to understand its limitations and use it appropriately to ensure your data is preserved. If your data source only retains 2 months of data and needs to retain 12 months of data in your reports, you may also need to consider other strategies, such as archiving your data to a location where Power BI can access it. This will provide a more robust solution and ensure that no data is lost when changing the model.

How to Get Your Question Answered Quickly

If it does not help, please provide more details with your desired output and pbix file without privacy information (or some sample data) .

Best Regards

Jianpeng Li

If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.

[Danfountain]

I started having a play with it and I think came up to the same conclusion but I am really glad you ratified it for me.  I appreciate there are better ways to do this but was looking for a temporary quick fix :-).  Back to the drawing board.  Thank you!

[v-jianpeng-msft]

Hi, @Danfountain 

Thank you very much for your reply. Whenever you need to do some deletion operations, it is a good habit to consider whether your data is lost or not. It would be a good practice if you were able to make some backups.

Best Regards

Jianpeng Li

[Danfountain]

Thank you for that.  So as far as I understand it - in your scenario I can delete data from the "archived data" and it will still show in the report but if i delete from the "incremental data" (last three days) then it would vanish.

If so this might be perfect for what i want to do 🙂