Forum Discussion
Incremental Refresh effect on Reference Table
I have set up a 'factTransaction' table, and used the 'sourcedate' for incremental refresh and 'moddateUTC' for the refresh changed data option.
I want to aggreagte the table further before loading (but after utilisitng the above incremental refresh parameters), so I was thinking creating a 'factTransAgg' table that references the 'factTransactions' table. Since the the reference happens further downstream than the incremental refresh parameters, would this work?
It should behave the same in your Dataflow; easy to confirm/try. I think it will work.
Pat
4 Replies
You can do still do a group by step that will fold after you filter with RangeStart and RangeEnd. Just add an aggregation to get the max of your moddateUTC column to use for change detection. Haven't confirmed but should work.
Pat
Hi Pat,
Sounds great, I understand this would probably work on the desktop app due to the manual nature of creating the range parameters, but would it work if I modified a dataflow after setting up the incremental refresh?
This is what led me to think about creating a reference table instead of modifying the source table.
It should behave the same in your Dataflow; easy to confirm/try. I think it will work.
Pat
Thanks Pat!
I can confirm that this works the way you outlined above.
