Full refresh + Incremental refresh of dataflows

[Anonymous]

Hi All,

We are doing an incremental refresh of one of the dataflow. The requirement is to do a full refresh once in a week and incremental refresh on rest of the days.
Right now the way we are achieveing it is by opening a dataflow, making some changes to metadata which forces a full refresh. But it's a time consuming and a very manual process.

Looking for suggestions. Thanks in Advance.

[arvindsingh802]

As incremental refresh will be keeping historical data then you should not be looking for full refresh in same dataflow

Can you help with your specific need to do full load on weekend so that I can suggest something if possible

If this post helps, then please consider Accept it as the solution and give it a thumbs up

[Anonymous]

We anticipate some changes in historical data in a week or two. Hence we are looking to find out if we can do a full refresh on weekends to avoid load on Service during week days. And continue with Incremental refreshes on weekdays.

[v-kkf-msft]

Hi @Anonymous ,

You can turn off incremental refresh in the settings.

Using incremental refresh with dataflows 

Or you can try to use the XMLA endpoint for incremental refresh. If a table has an incremental refresh policy defined, applyRefreshPolicy will determine if the policy is applied or not. If the policy is not applied, a process full operation will leave partition definitions unchanged and all partitions in the table will be fully refreshed.

However, both methods require manual operation.

Best Regards,
Winniz

If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.