Forum Discussion
Data flow incremental refresh with backfill
I read it's best to create a data flow which is then incrementally refreshed every x days. On top of this 1 or more semantic models can then be created and incrementally refreshed as neededWhere did you read that?
Dataflow incremental refresh is a black box. You cannot bootstrap anything and you cannot refresh individual partitions.
Having both the dataflow and the semantic model do incremental refresh is a waste of time and storage.
Drop the dataflow and load the data directly into the semantic model. Use bootstrapping and XMLA to fill the partitions. Each partition needs to be less than 10 GB and needs to refresh within 5 hours, so aim for a safe size of about 5GB maximum per partition or 2.5 hours refresh (whatever is the limit that hits first).
I read it's best to create a data flow which is then incrementally refreshed every x days. On top of this 1 or more semantic models can then be created and incrementally refreshed as needed
Where did you read that?
Dataflow incremental refresh is a black box. You cannot bootstrap anything and you cannot refresh individual partitions.
Having both the dataflow and the semantic model do incremental refresh is a waste of time and storage.
Drop the dataflow and load the data directly into the semantic model. Use bootstrapping and XMLA to fill the partitions. Each partition needs to be less than 10 GB and needs to refresh within 5 hours, so aim for a safe size of about 5GB maximum per partition or 2.5 hours refresh (whatever is the limit that hits first).