How to stage ingestion of 10 years of data with dataflows ?

[frano72]

Hi,

I"m still struggling on finding a pattern for ingesting many years of data and then keeping that refreshed day by day.  The issue is that i want to stage the ingestion, as the SQL stored procedure that runs to generate the 10 years of data takes a looonng time.  Don't want to have separate stored procs for date ranges and have separate dataflows either.

I tried doing two dataflow refreshes and changing dates in the stored procedure in between but the data got overwritten (had incremental refresh on) each time.  my dates were at the start of my set ie 2010, so am guessing incremental refresh has no effect in this situation as that would be working from current dates and looking backwards from today...

Any ideas...

[v-lid-msft]

Hi @frano72 ,

Based on my research, Once incremental refresh is configured, the dataflow automatically alters your query to include filter by date as your test result, for example set for store data for ten years, it means store data of past ten years from today.

We can use the selectrows function to keep all the data from given day, but it is a full-size ineffective refresh.

Best regards,

Community Support Team _ Dong Li
If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.