Forum Discussion
How to not overwrite data in Power BI Dataflow
What prevents you from implementing incremental refresh against the original data source?
Hello!
Apologies fir the late response. Actually what happens is that the original datasource is a connection through a REST API that will return the last 4 months of data.
What happens is that we get this data through 3 different dataflows (one for each Region because if not is too heavy to get the data and it is closed due to a timeout).
Basically the requisite is that, as this data is continuously changing, it is needed to aggregate all the snapshoots we do for 2 weeks. As an example, yesterday I loaded data and I save yesterday's date in a column (02/14/2022). Today I will load new data and I need to append it after what I got yesterday but with today's date (02/15/2022).
My problem is that I am not being able to append this data as when the append occurs it overwrittes whatever was already there because the "source" table does not have yesterday's information anymore, only today's one.
My question still stands. What prevents you from implementing incremental refresh against the original data source?
What prevents me is that, as data comes from a query to an API, the original data source is overwritten with new data on every load and in order to use an incremental refresh, "old" data needs to persist.
The data that I had yesterday, today is erased and changed with today's data.You may be better off using event based reporting. Are your data changes tracked in your source system?