How to manage data restatements?

[Charlie_Miller]

I receive data on a quarterly basis that goes back in time 12 months. There are changes or restatements made to the old data. Simply put, I get 9 months of restated data plus a new quarter of data. 

So for example, I currently have data from Q1 2020 - Q1 2024. I just received a data drop which includes new data for Q2 2024 but it also contains restated data for Q3 2023 - Q1 2024. 

How can I best handle refreshing my power BI report? Right now I would filter out the most recent 3 quarters from my query and then append it with the new data drop. I am worried about harming performance and am wondering if I can go about this in a better way, especially as I will have to repeat this process every quarter. Any advice is much appreciated. 

[Anonymous]

You need to just bring in and append each of the last 4 quarters. It's simply not true at all that Power BI/Power Query is not prepared to handle this scenario. Many of us append millions and millions of rows of data as a matter of course, with perfectly fast performance. Further, if you set up your query to return just the last 4 quarters, I don't see why you'd need to repeat this every quarter. 

--Nate

[Charlie_Miller]

But my issue is that it isn't as simple as just appending data to my existing data. If I do that, I will be adding duplicates. Every quarterly data drop includes a new quarter of data but it also includes restated data for the previous 3 quarters making updates to the data previously distributed. 

Not sure if I am articulating this well, but I cannot just append each quarterly data drop to the existing data. I have to replace existing data as well. 

[lbendlin]

Power BI is fundamentally unprepared for handling this.  You need to do all the work yourself, either further upstream, or by implementing SCD2 handling etc. 

If your data source is covered by incremental refresh then you can choose to refresh the affected partitions selectively, or choose a larger "hot" window.