Forum Discussion
Incremental refresh -Detect Data Change column vs Partition Column
Yes the documentation says to use 2 different columns, one for the incremental refresh and another for the query filter parameters. However, what actually happens if you watch the sql it generates is the incremental refresh column (e.g. modified time) gets used only for detecting if there are changes to pickup in the time period you specify (such as the last 1 day). It then totally ignores that column when it comes to actually pulling data. You read that right - it's ignored after that.
Case in point:
I have an order date on a fact table that I want to partition on (but not incremental refresh since other things can change other than the order date) so I put the order date as the parameter filter column. I have an audit timestamp I want to use for incremental refresh so I put that in the incremental properties for detecting changes.
But when an incremental refresh runs it does the following 2 queries:
1) "select max([rows].[audit_tms]) from....subquery. It's just finding out if there's something to update. The incremental refresh column is not used again for querying data.
2) Select the data based on the parameter query filter for that time period. In this case the order date. (It goes into the partition that aligns with this column's data).
End result: I leak 100% of any changes that do not cause the order date to change.
If however you use the audit timestamp for both incremental refresh AND the query filters (against their advice) then you won't leak data BUT you will end up with a few bloated partitions. If you do complete reload then you will end up with only current partition with 100% of your data. Over time that will start to spread out...a little.
So you have 2 choices:
1) Leak data changes (if you follow their advice)
2) Lose most of the performance advantage of partitioning
Bottom line is this product's architecture if fundamentally flawed. It seems to me that they built the tool with the assumption that you only insert and not update. Or if you do update you also update the business date. I think there may be some cases you have facts where this is true but I'm thinking that's fairly rare. Where I work nothing works this way - everything can be updated often due to re-alignment of keys such as customer assignments (due to merges, householding, etc) or simply an enhancement that requires a reload.
What Microsoft should have done is allow you to pull based on one column but partition based on an alternate static business column. I'm just baffled they didn't design it that way.