Large Dataset -Incremental Refresh

[Learner_SG]

Hi ,

I have a dataset which is growing day by day. Previously , I was able to run the refresh without much delay. But now it seems to take longer time to run. I am trying to change it into incremental refresh , still seems to take longer time. What are the possible ways to correct it?

Could anyone help?

[v-zhangti]

Hi, @Learner_SG 

Perhaps you could try dataflow and incremental refresh in conjunction.

A good alternative is incremental refresh, which provides the following benefits for dataflows:

• Refresh occurs faster: Only data that's changed needs to be refreshed. For example, refresh only the last five days of a 10-year dataflow.
• Refresh is more reliable: For example, it's not necessary to maintain long-running connections to volatile source systems.
• Resource consumption is reduced: Less data to refresh reduces overall consumption of memory and other resources.

Please check the following document, we hope it can help you.

https://docs.microsoft.com/power-query/dataflows/create-use 

https://docs.microsoft.com/power-query/dataflows/incremental-refresh 

Best Regards,

Community Support Team _Charlotte

If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.

[Learner_SG]

@v-zhangti my source data is azure cosmos db and to use dataflow what I understand is that we need to use gateways and the company does not want to use the gateway approach. So ,what else could be done?any inputs.thanks.

[PC2790]

Review your parameters set for incremental refresh once.

Here, refer this and see if it turns out to be helpful.