Skip to main content
cancel
Showing results for 
Search instead for 
Did you mean: 

Find everything you need to get certified on Fabric—skills challenges, live sessions, exam prep, role guidance, and more. Get started

Reply
lg01
Frequent Visitor

Datamart Incremental refresh logic

I am trying to set up incremental refresh in some tables of my datamart. The documentation says that the incremental refresh is basically a "partition" of the window we want to refresh and this window is constally rolling forward based on the date and it will drop the oldest record. So I need to clarify my interpretation. If I choose to maintain 5 years of history, and refresh records from the past year, does that mean it will always maintain  5 years of data to the date and truncate the most recent year and reload it including all the new records?

I see a field that says "Detect data changes"; but I actually don't want it to detect changes in individual records. I want it to just reload my most recent year completly. Is that possible?

1 ACCEPTED SOLUTION
lbendlin
Super User
Super User

does that mean it will always maintain  5 years of data to the date and truncate the most recent year and reload it including all the new records?

Mostly correct. In reality it will do full years  (partitions are defined by regular calendar boundaries).  So for a while you may end up with six partitions instead of five.

 

Stay away from the "Detect data changes" setting.  It will double your storage requirements.  You should know when older partitions need to be refreshed, and you should decide when to do that.

View solution in original post

3 REPLIES 3
lbendlin
Super User
Super User

does that mean it will always maintain  5 years of data to the date and truncate the most recent year and reload it including all the new records?

Mostly correct. In reality it will do full years  (partitions are defined by regular calendar boundaries).  So for a while you may end up with six partitions instead of five.

 

Stay away from the "Detect data changes" setting.  It will double your storage requirements.  You should know when older partitions need to be refreshed, and you should decide when to do that.

lg01
Frequent Visitor

Thanks for your reply! That is great to know. How can I see these partitions? 

That would be nice, wouldn't it?  But datamarts are based on dataflows which are based on csv/parquet files.  There is no known (to me) way to access these partitions directly, neither for dataflows nor for datamarts.

Helpful resources

Announcements
July 2024 Power BI Update

Power BI Monthly Update - July 2024

Check out the July 2024 Power BI update to learn about new features.

July Newsletter

Fabric Community Update - July 2024

Find out what's new and trending in the Fabric Community.