Forum Discussion
Is it possible to implement Incremental Refresh based upon Last Modified Date?
Hi, I want to implement Incremental Refresh for my Power BI Dataset but all the videos/guides I have watched/read apply incremental refresh based upon a Date Range given by themselves.
I was wondering whether is it possible to do the same using Last Modified Date in my Fact Table. I have a field in Fact Table that tells me when was a certain row modified.
Now, I was wondering that if I could maybe get the Timestamp for "Last Refresh" and then only refresh those rows whose modified date was greater than the Last Refresh time. Does anybody know whether such a thing is possible or not?
Hey Anonymous ,
it's possible but not that simple. The reason for that is, incremental refresh means new data will be added. The database engine that is powering a Power BI dataset supports table partioning. A partion is described by a start and end value, often these values are datetime data types but not necessarily have to. If you are familiar with data warehousing than it's necessary to understand that the database engine does not do upserts, instead it does partition purge and partition load.
By default Power BI's incremental refresh supports a modified date, this article explains the intricate workings of Power BI's incremental refresh: https://learn.microsoft.com/en-us/power-bi/connect-data/incremental-refresh-overview?WT.mc_id=DP-MVP-5003068
I also recommend reading everything related about incremental refresh by the Tabular Editor team, starting here: https://docs.tabulareditor.com/te3/tutorials/incremental-refresh/incremental-refresh-setup.html.
Hopefully, this gets you started
Tom
1 Reply
Hey Anonymous ,
it's possible but not that simple. The reason for that is, incremental refresh means new data will be added. The database engine that is powering a Power BI dataset supports table partioning. A partion is described by a start and end value, often these values are datetime data types but not necessarily have to. If you are familiar with data warehousing than it's necessary to understand that the database engine does not do upserts, instead it does partition purge and partition load.
By default Power BI's incremental refresh supports a modified date, this article explains the intricate workings of Power BI's incremental refresh: https://learn.microsoft.com/en-us/power-bi/connect-data/incremental-refresh-overview?WT.mc_id=DP-MVP-5003068
I also recommend reading everything related about incremental refresh by the Tabular Editor team, starting here: https://docs.tabulareditor.com/te3/tutorials/incremental-refresh/incremental-refresh-setup.html.
Hopefully, this gets you started
Tom