Starting December 3, join live sessions with database experts and the Microsoft product team to learn just how easy it is to get started
Learn moreGet certified in Microsoft Fabric—for free! For a limited time, get a free DP-600 exam voucher to use by the end of 2024. Register now
Hi All
I wanted to understand how the rows which are deleted from table are handled in incremental refresh dataset.
Does it identify the deleted rows and delete from dataset OR Do we need to do full refresh in PBI desktop and then publish?
Thanks & regards
Harsh Dhawan
I am not sure as it depends on your data!
Key columns: Created date, ETL Date, IsDeleted Flag.
Insert, Update, Delete is possible in any partition at any point
Table to be incrementally refreshed is a fact table
I have access to make database modificaitons if required.
Right now the only solution I can think of is to include - where IsDeleted = False, in the RLS logic for all our RLS groups.
Thanks Gilbert for your reply
I have a table which contains 12 million rows. and it is modelled based on row level permissions from source system
This table is truncated and loaded on daily basis, Now I am being asked to look at the possibility to do incremental refresh for dataset which is loading this table
In case user permissions are removed from any record in source table, that data rows will not be loaded in the next etl run
I am not sure how that row will be removed from power bi dataset during the next incremental refresh if it no longer exists in source table now
Hope my question makes sense!
Cheers
Handling deletes is just what I am after. Thanks for posting that link 🙂
The approach outlined by Patric is not ideal but it's the next best thing - until that capability is added to incremental refresh.
Hi @GilbertQ Patrick's Method in the link doesn't seem to be working. The polling expression folds the query with the delete filter back to the source. This means that the max(lastupdateddate) queries will exclude deleted rows, hence that partition will not be updated.
Do you know of any way to handle this?
The work around I used was to allow all the deleted lines and manage them using Row level security. This was the only was I saw to manage the issue.
Starting December 3, join live sessions with database experts and the Fabric product team to learn just how easy it is to get started.
March 31 - April 2, 2025, in Las Vegas, Nevada. Use code MSCUST for a $150 discount! Early Bird pricing ends December 9th.
User | Count |
---|---|
33 | |
32 | |
19 | |
12 | |
8 |
User | Count |
---|---|
52 | |
37 | |
29 | |
14 | |
12 |