Incremental refresh based on two date fields

[Anonymous]

Hi,

I have a report build with data created on or after Jan 1,2019 or updated on or after Jan 1,2019.

"Created on" and"Updated on" are my two date fields. Can someone please let me know how I could do a load which only brings the changed records and new records without refreshing the entire data.

Thanks

Poojitha

[V-lianl-msft]

Hi @Anonymous ,

You can learn how to set up incremental refresh through this blog:https://community.powerbi.com/t5/Community-Blog/Important-Considerations-for-Setting-up-Incremental-Refresh-on/ba-p/554151 

The first refresh may take longer to import the historical data. Subsequent refreshes can be much quicker because they use incremental refresh.

Best Regards,
Liang
If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.

[Anonymous]

My source does not support query folding, when I set up the incremental refresh, it is always in "In Progress" state and never completes.

Please suggest.

Thanks

[V-lianl-msft]

Hi @Anonymous ,

If the data source supporting query folding, we just get that part of the data and load it.

If the data source doesn’t support folding, we have to read the entire data anyways, load it into temporary memory, then do filtering and load only the last one year. Then the incremental load won’t make the refresh process that much different, because the data will be loaded fully from the source to the engine of Power Query every time, and filtered there.

One way is to create a dataflow for previous periods manually, run it once. and then create the rest in a new dataflow.

Best Regards,
Liang
If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.

[Anonymous]

Hello,

I am basically working with the incident management data where the source is ServiceNow. So, there is a good chance that the pervious period data is modified in the future, such cases where the incidents are open for months, Hence, I am not sure if two data flows will work.

Also, Now I see that after a long period of time after the incremental refresh was run, I get a error message as below:

 

Could you please suggest

Thanks

Poojitha