Dynamic Power BI Refresh for set of records at the same time old records should remain there

[Anonymous]

I am working on a use case where we have an ETL responsible for pulling electric equipment data every month from different countries worldwide. The load frequency is daily, but monthly data for each country is loaded from the source to my application.

I have a Power BI reporting layer and its associated semantic models which pull data from my application. So, we would want to have a capability in reporting where monthly data for each country should remain available until the next month-end has not started. At the same time, the source table has data for other clients that should be there in Power BI, but due to high volume, I do not want to do anything with old data. Only new data for each client should be deleted and reloaded with the latest data. How can we achieve this with daily refresh?

[Anonymous · Accepted solution]

Hi @Anonymous ,
Depending on the questions you ask and the goals you want to achieve, you can try to manage your data using incremental refreshes so that you can refresh only the latest data (e.g., the current month's data for each country/region) while retaining historical data without having to reload the data. You can define the time period during which new data should be loaded (e.g., daily for the month) and set retention policies for historical data. Of course, you can also partition the data by month and country/region in the application database. This method, combined with incremental refreshes, can increase the efficiency of the refresh process by targeting only the partitions that need to be updated or replaced with new data.
Incremental refresh for semantic models and real-time data in Power BI - Power BI | Microsoft Learn
Configure incremental refresh and real-time data for Power BI semantic models - Power BI | Microsoft Learn
Advanced incremental refresh and real-time data with the XMLA endpoint in Power BI - Power BI | Microsoft Learn

Best regards,

Albert He

If this post helps, then please consider Accept it as the solution to help the other members find it more quickly