Forum Discussion
Incremental refresh selective outage
I have two folders in sharepoint called f1 (receives daily files) and f2 (receives daily files)
-> folder contents of f1 (68 files) and f2 (100 files) respectively go into datflows called df1 and df2. df1 and df2 are working as desired.
-> df1 and df2 are called into a dataset(ds) to build 2 seperate tables called t1 and t2.
The incremental refresh has been configured for t1[date Modified] and t2[date Modified] with exact same condition.
{
........
"rollingWindowGranularity": "year",
"rollingWindowPeriods": 10,
"incrementalGranularity": "day",
"incrementalPeriods":1,
"pollingExpression": .....
.......
}
However, t1 does not bring all the data from df1 after incremental. I expect to see 68 files; but I see only 67 files where as for t2 it returns all the 100 files.
The funny thing is, if my df1 has 80 files today and I build ds today with incremental refresh on both t1 and t2. It will show all the 80 files. But if I check the ds resepectively 1 from today (80+1), 2 day from today (80+2), 3 day from today (80+3) .........I only see the initial 80 files as part of t1 that were there on the initial refresh.
Do you know what might be causing this? Cause df1 is updating daily and has all the files as desired (80+1), (80+2), (80+3)...(80+n). Why t1 of ds are not getting updated with (80+n).
t1 is simply calling df1 and there is aboslutely no transformation involved (apart from incrementalRefresh Filtering)
4 Replies
- GilbertQ
Super User
Hi smpa01
What I have found is that because all the Power BI Servers are using UTC for their datetime, what can sometimes happen is when the incremental refresh is running it will not include your current date. And that is why it appears to be missing the data.
To solve this I always change the incremental refresh policy to refresh the last 2 days to ensure that it always gets the data irrespective of time. - edhans
Community Champion
I'm not clear. Incremental Refresh only works against sources that can accept query folding. SharePoint Folders is not one of them. So not sure how you even published the config for IR for data coming from a SharePoint folder.
- smpa01
Community Champion
Incremental works for sharepoint folder contents.