Forum Discussion

lg01's avatar
lg01
Advocate III
2 years ago
Solved

Datamart Incremental refresh logic

I am trying to set up incremental refresh in some tables of my datamart. The documentation says that the incremental refresh is basically a "partition" of the window we want to refresh and this windo...
Need Help
  • lbendlin's avatar
    lbendlin
    2 years ago
    does that mean it will always maintain  5 years of data to the date and truncate the most recent year and reload it including all the new records?

    Mostly correct. In reality it will do full years  (partitions are defined by regular calendar boundaries).  So for a while you may end up with six partitions instead of five.

     

    Stay away from the "Detect data changes" setting.  It will double your storage requirements.  You should know when older partitions need to be refreshed, and you should decide when to do that.

lbendlin's avatar
lbendlin
Super User
2 years ago
does that mean it will always maintain  5 years of data to the date and truncate the most recent year and reload it including all the new records?

Mostly correct. In reality it will do full years  (partitions are defined by regular calendar boundaries).  So for a while you may end up with six partitions instead of five.

 

Stay away from the "Detect data changes" setting.  It will double your storage requirements.  You should know when older partitions need to be refreshed, and you should decide when to do that.

lg01's avatar
lg01
Advocate III
2 years ago

Thanks for your reply! That is great to know. How can I see these partitions? 

  • lbendlin's avatar
    lbendlin
    Super User
    2 years ago

    That would be nice, wouldn't it?  But datamarts are based on dataflows which are based on csv/parquet files.  There is no known (to me) way to access these partitions directly, neither for dataflows nor for datamarts.