Forum Discussion

Anonymous's avatar
Anonymous
Not applicable
5 years ago

Incremental Refresh - Combine History load (SharePoint online) with Daily load(via API Calls)

jeffshieldsdev TomMartens v-syr v-easonf-msft  PhilipTreacy amitchandak Greg_Deckler amitchandak  Requirement is to to combine 5 years of History data(Share Point Online) and append it with last 7 ...
Dataflow
Need Help
Anonymous's avatar
Anonymous
Not applicable
5 years ago

Hi Anonymous 

How is your incremental refresh set ? Could you provide a screenshot of the corresponding settings ?

(1)In incremental refresh , there are two time-related settings . Store rows in the last and Refresh rows in the last .

Store rows in the last : this required setting determines the historical period in which rows with a date/time in that period are included in the dataset, plus rows for the current incomplete historical period, plus rows in the refresh period up to the current date and time.

Refresh rows in the last : this required setting determines the incremental refresh period in which all rows with a date/time in that period are included in the refresh partition(s) and refreshed with each refresh operation.

So you should make sure that your storage and refresh cycle settings are correct .

(2) Regarding the problem of repetitive refresh, could you change the time period for the next refresh to make sure the refresh time is staggered ?

(3) For data sources supported by incremental refresh ,you can refer to the link

Incremental refresh for datasets in Power BI - Power BI | Microsoft Docs

 

Best Regards

Community Support Team _ Ailsa Tao

 

If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.

Anonymous's avatar
Anonymous
Not applicable
5 years ago

Hi Anonymous 

Thanks for your response , was not sure if i would ever get a response to my query . 

Regarding the two dates , i have ensured the number mentioned in store rows fields includes my historical data and refresh rows is set for 7 days , the reason why i chose to refresh 7 days is there might be some backdated transactions and we wanted to reload the data for last 7 days agian  but setting up this way in dataflows creates duplicate data . i can set this up for 1 day to test this out . 

i have shared the screenshot of 

 

here are some of the key configurations from our side :

  1. We are currently on power bi premium subscription
  2. We would need to use dataflows (as all transformation logic is stored in dataflows as well) to setup incremental refresh , as this dataflow would be used by multiple reports within individual workspaces catered to end users
  3. About data –
    1. Historical data : historical (one off activity) data is delivered by our DWH vendor through sftp which we are storing into SharePoint online and pointing our Power BI to this location
    2. Our daily data is to be loaded using API calls to our DWH vendor location ( which is stored in US Data centre)
    3. Key thing to also note is that Daily data is only available on weekdays only

 

Hope this information is useful for you to get an idea of our current setup.   Keen to hear your views