Forum Discussion
Incremental Refresh - data being removed from source
Hi all,
QQ. With incremental refresh - if you had a source that was only keeping 2 months data, could incremental refresh be used to keep 12 months data within the report?
Many thanks
Dan
Hi, Danfountain
If you want to change a semantic model that has been configured for incremental refresh, such as a full refresh of your semantic model, you will lose the data in the Archive data. You'll need to re-upload your PBIX file in Power BI Desktop. You can refer to the case link below:
Solved: How to clear or empty out a Power BI dataset that ... - Microsoft Fabric Community
Incremental refresh for Power BI refreshes are based on the policies that you set. In incremental refresh, when you set an incremental refresh policy, Power BI creates and manages partitions of your data. Only the most recent data that has changed needs to be refreshed. If you have successfully incrementally refreshed several times. If you suddenly have only nearly two months of data in your data source one day, the data in the Archive data compartment will still be able to use that data in Power BI, but it will be phased out according to the policies you set.
While incremental refresh is a powerful feature, it's important to understand its limitations and use it appropriately to ensure your data is preserved. If your data source only retains 2 months of data and needs to retain 12 months of data in your reports, you may also need to consider other strategies, such as archiving your data to a location where Power BI can access it. This will provide a more robust solution and ensure that no data is lost when changing the model.
How to Get Your Question Answered Quickly
If it does not help, please provide more details with your desired output and pbix file without privacy information (or some sample data) .
Best Regards
Jianpeng Li
If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.
6 Replies
Hi, Danfountain
Incremental refresh based on the policy you set, and here are some detailed explanations:
The following diagram shows the entire workflow for incremental refresh:The first refresh loads new and updated data from the refresh period, as well as historical data for the entire storage period.
1.Archive data:
In this example, Power BI deletes data from 5 years prior to today (11/17/2022). Load and save data for 1/1/2017-11/17/2022. If your data is in this range, then they are still valid.
2.Incremental data
In this example, Power BI Service refreshes the data for the last three days today (11/20/2022) 11/18/2022-11/20/2022. If you delete this part of the data, the changes will be reflected in your semantic model.
Here are some documents that provide insight into incremental refresh:
Incremental refresh for semantic models and real-time data in Power BI - Power BI | Microsoft Learn
A deep dive into Power BI incremental refresh feature | by Jonathan Law | Towards Data Science
Aggregated Intelligence: Understanding PowerBi Incremental Refresh with Detect Changes
Handling of Deleted data in incremental refresh - Microsoft Fabric Community
Solved: How to clear or empty out a Power BI dataset that ... - Microsoft Fabric Community
How to Get Your Question Answered Quickly
If it does not help, please provide more details with your desired output and pbix file without privacy information (or some sample data) .
Best Regards
Jianpeng Li
If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.
Thank you for that. So as far as I understand it - in your scenario I can delete data from the "archived data" and it will still show in the report but if i delete from the "incremental data" (last three days) then it would vanish.
If so this might be perfect for what i want to do 🙂
I have now found another question.
So imagine i have done the above. The data refreshes nightly and the "deleted" data is preserved in powerBI.
Now lets just say a few months pass and its running. I then want to make some changes. As soon as I try to change the model its no longer going to be able to get that data and therefore I will lose it? If so this might not be the option that i thought it was.
Hi, Danfountain
If you want to change a semantic model that has been configured for incremental refresh, such as a full refresh of your semantic model, you will lose the data in the Archive data. You'll need to re-upload your PBIX file in Power BI Desktop. You can refer to the case link below:
Solved: How to clear or empty out a Power BI dataset that ... - Microsoft Fabric Community
Incremental refresh for Power BI refreshes are based on the policies that you set. In incremental refresh, when you set an incremental refresh policy, Power BI creates and manages partitions of your data. Only the most recent data that has changed needs to be refreshed. If you have successfully incrementally refreshed several times. If you suddenly have only nearly two months of data in your data source one day, the data in the Archive data compartment will still be able to use that data in Power BI, but it will be phased out according to the policies you set.
While incremental refresh is a powerful feature, it's important to understand its limitations and use it appropriately to ensure your data is preserved. If your data source only retains 2 months of data and needs to retain 12 months of data in your reports, you may also need to consider other strategies, such as archiving your data to a location where Power BI can access it. This will provide a more robust solution and ensure that no data is lost when changing the model.
How to Get Your Question Answered Quickly
If it does not help, please provide more details with your desired output and pbix file without privacy information (or some sample data) .
Best Regards
Jianpeng Li
If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.
