Forum Discussion
Handling of Deleted data in incremental refresh
Super UserWhen using Incremental refresh it is done in the backend in the Power BI Service where the model is hosted.
The process actually drops the partitions which are considered to be outside of the data window of data to keep.
Thanks Gilbert for your reply
I have a table which contains 12 million rows. and it is modelled based on row level permissions from source system
This table is truncated and loaded on daily basis, Now I am being asked to look at the possibility to do incremental refresh for dataset which is loading this table
In case user permissions are removed from any record in source table, that data rows will not be loaded in the next etl run
I am not sure how that row will be removed from power bi dataset during the next incremental refresh if it no longer exists in source table now
Hope my question makes sense!
Cheers
- GilbertQHi there
If I understand your question correctly let me try and explain it by like this.
When using incremental refreshing in Power BI, older data will not be refreshed or imported again into Power BI.
Based on your explanation above that would mean that certain data must be deleted or changed because the underlying data has changed.
You have 2 options, the first one is to just reload all the data. I currently have a project where I am uploading 15 million rows and it refreshes in about 20minutes.
Option number 2 is that you can use the detect data changes when using Incremental refresh with Power BI. Here is a great video explaining how to do it by GuyInaCube: https://www.youtube.com/watch?v=JsJWBr1_ktQHandling deletes is just what I am after. Thanks for posting that link :)
The approach outlined by Patric is not ideal but it's the next best thing - until that capability is added to incremental refresh.