Forum Discussion

smpa01's avatar
smpa01
Icon for Community Champion rankCommunity Champion
3 years ago

Incremental refresh selective outage

I have two folders in sharepoint called f1 (receives daily files) and f2 (receives daily files)

        -> folder contents of f1 (68 files) and f2 (100 files) respectively  go into datflows called df1 and df2. df1 and df2 are working  as desired.

        -> df1 and df2 are called into a dataset(ds) to build 2 seperate tables called t1 and t2.

 

The incremental refresh has been configured for t1[date Modified] and t2[date Modified] with exact same condition.

 

 

{
........
"rollingWindowGranularity": "year",
"rollingWindowPeriods": 10,
"incrementalGranularity": "day",
"incrementalPeriods":1,
"pollingExpression": .....
.......

}

 

 

 

However, t1 does not bring all the data from df1 after incremental. I expect to see 68 files; but I see only 67 files where as for t2 it returns all the 100 files.

 

The funny thing is, if my df1 has 80 files today and I build ds today with incremental refresh on both t1 and t2. It will show all the 80 files. But if I check the ds resepectively 1 from today (80+1), 2 day from today (80+2), 3 day from today (80+3) .........I only see the initial 80 files as part of t1 that were there on the initial refresh.

 

Do you know what might be causing this? Cause df1 is updating daily and has all the files as desired (80+1), (80+2), (80+3)...(80+n). Why t1 of ds are not getting updated with (80+n).

 

t1 is simply calling df1 and there is aboslutely no transformation involved (apart from incrementalRefresh Filtering)

 

 

4 Replies

  • Hi smpa01 

     

    What I have found is that because all the Power BI Servers are using UTC for their datetime, what can sometimes happen is when the incremental refresh is running it will not include your current date. And that is why it appears to be missing the data.


    To solve this I always change the incremental refresh policy to refresh the last 2 days to ensure that it always gets the data irrespective of time.

  • edhans's avatar
    edhans
    Icon for Community Champion rankCommunity Champion

    I'm not clear. Incremental Refresh only works against sources that can accept query folding. SharePoint Folders is not one of them. So not sure how you even published the config for IR for data coming from a SharePoint folder.

    • smpa01's avatar
      smpa01
      Icon for Community Champion rankCommunity Champion

      Incremental works for sharepoint folder contents.