Forum Discussion
Handling of Deleted data in incremental refresh
Thanks Gilbert for your reply
I have a table which contains 12 million rows. and it is modelled based on row level permissions from source system
This table is truncated and loaded on daily basis, Now I am being asked to look at the possibility to do incremental refresh for dataset which is loading this table
In case user permissions are removed from any record in source table, that data rows will not be loaded in the next etl run
I am not sure how that row will be removed from power bi dataset during the next incremental refresh if it no longer exists in source table now
Hope my question makes sense!
Cheers
Super UserIf I understand your question correctly let me try and explain it by like this.
When using incremental refreshing in Power BI, older data will not be refreshed or imported again into Power BI.
Based on your explanation above that would mean that certain data must be deleted or changed because the underlying data has changed.
You have 2 options, the first one is to just reload all the data. I currently have a project where I am uploading 15 million rows and it refreshes in about 20minutes.
Option number 2 is that you can use the detect data changes when using Incremental refresh with Power BI. Here is a great video explaining how to do it by GuyInaCube: https://www.youtube.com/watch?v=JsJWBr1_ktQ
Handling deletes is just what I am after. Thanks for posting that link :)
The approach outlined by Patric is not ideal but it's the next best thing - until that capability is added to incremental refresh.
- GilbertQYeah for sure, I am hopeful that over time the incremental refresh story will improve.
Hi GilbertQ Patrick's Method in the link doesn't seem to be working. The polling expression folds the query with the delete filter back to the source. This means that the max(lastupdateddate) queries will exclude deleted rows, hence that partition will not be updated.
Do you know of any way to handle this?
