Forum Discussion

Yaa_D's avatar
Yaa_D
Frequent Visitor
4 years ago

Incremental Refresh and dataflows to save historic data

Hi All,
I'm trying to use dataflows and incremental refresh to get and save data from an external API.

I've created a dataflow (A) where I'm accessing the API to pull the most recent data, this dataflow is refreshed once a week.
I've created another dataflow (B) using dataflow A as it’s source, and set up incremental refresh to refresh data from the past 2 days and keep data for 16 months. This refresh occurs once a week, on the following day to dataflow A's refresh.

The problem is: data flow B keeps being overwritten with the most recent data a from dataflow A, the previous data is not saved at all.

I'm not sure why this doesn't work, maybe it's related to dataflow A being a 'linked entity' in dataflow B, and then referenced?

I have Incremental refresh working in a completely different dataflow, so I thought I had it figured out, obviously I was wrong 🙂

Should this work? not sure if there is something I'm missing in setting up these dataflows, or if I need to completely change my approach?

Thank you for your help!

1 Reply