Hello guys,
i'm implementing incremental refresh in my dashboard and i'm facing some questions about partitionning and how it works.
in my DWH i have multiple fact table that i want to set incremental refresh on.
For each of this table i have a creation_date which i use to determine the range period for history.
i have also toggled on the "detect data changes" option based on another column update_date. I think i'm following the right path for the moment.
if i understood correctly how "detect data changes" works, after each refresh it stores the max update_date for each partition, and on the next refresh, it compares the max date of each partition to see if there is a new max date. And it refresh that partition if there is a new max date.
the question i have is about partitionning. I've open my dataset with tabular editor and saw all the partitions created automatically after the initial refresh (from daily for the firsts month to yearly for the far years).
My dataset in composed of data from multiple clients. And i will be adding more client's data in the future if a new client ask for it.
I don't understand how partition will be created. Are they based on the creation_date available in my DWH ?
If it is the case, the first time i insert a new client in the dataset, all his data (from all fact table) will have the same creation_date. So all the initial data from one client will be on the same partition ?
And let say a few days later a client make a single change to his initial data, so only one row is changing somewhere, the max update_date for that partition will change and i will refresh all the data from that client just because of one line change ? i don't really consider this as incremental refresh because it will load all data all the time instead of only the data that are changing.
Should i create custom partitions to avoid reload all client data if only one row is changing ?
May be i'm completely wrong in my understanding of the incremental refresh and partitionning.
Thanks a lot for your help
