Incremental Refresh using DataFlow - Loading historical data by partition

[holypond2001]

Hi, I'm in a bit of weird situation where my source data cannot run a full year's worth of data.

It will timeout, so I need the source data to pull 3 months worth of data for each load.

To load all historical data into PBI report starting Jan 2022 to Aug 2023, I'm trying to set up incremental refresh.

My thought is, my incremental refresh policy will start as "refresh rows from the past 17 months" to load Mar-May (since I would already have loaded Jan-Mar as my initial run).

Then, switch the source data to Jun-Aug and update the policy to "refresh rows from the past 14 months" to load the new 3 months and so forth, while archiving previously loaded months.

Will this work or does changing the incremental refresh policy reset everything?

Also this is really PIA for me to keep changing up the policy and refresh 4 times just to load 12 months. Please advise if there's any better idea... Thank you!

[sergej_og]

Hmm, I don´t use OData connector.
But http 429 sounds like you send too much requests to your source.
I know there are limitations for any API calls depends on your contract with your source provider (Dayforce).

Thats all I can say at the moment from your messages.
Do you have restricted API calls per minute, hour etc.?
Maybe this can cause an issue.

Regards

[sergej_og]

Hey @holypond2001 ,
what is your approach while loading data from source? Just one whole table or a SQL Statement sending to source?
I´m not sure about your idea to load date in 3-months-steps.
Incremental refresh for dataflows is possible (not for every source!) by using this option:
In your loaded table(s) you can see this options: marked one is for incr. refresh

But I think your bigger trouble is timeouting.
Try to fix it first.

Regards

[holypond2001]

Hello @sergej_og 

My source data is Dayforce (emp managment app) and I'm connecting to its report via OData.

Another issue I'm running into now is when I do a full refresh (for 3 month periods) it refreshes fine. But when I set up the incremental refresh, it errors out with code 429 "Too many requests".

Any idea why it's behaving this way?