Forum Discussion
Helper IIncremental Refresh with data for less than a day
So I know about how to enable incremental refresh in Power BI, I have it set to store 1 year and refresh last 1 day.
The issue I am having is because the data source is large it can reliably return only 2 hours of data at a time. So I can return data from 4 hours ago but upto 2 hours ago. So it means if we take current time i can not fetch last 2 hours and can only fetch the previous 2 hours.
I do not understand one thing, will this low amount of hours be able to even incrementally refreshed. Am I to expect that at the end of the day all such refreshes will add upto a day? And also the settign for live data apparently will show only last 2 hours instead.
So in a nutshell i am asking how does incremental refresh work with data which only holds a few hours in it? I have played around and reliably tested the data which arrives for entire 3 days back with this i see no issues right now.
Thanks for your help.
8 Replies
- mohammedadnant
Impactful Individual
Hi dkalinai
This is a bit strange,
As far as I know, it all depends on the parameter that we select, in your case you can set things as per the image below, I believe you have the premium capacity to get the latest data using the direct query.
https://docs.microsoft.com/en-us/power-bi/connect-data/incremental-refresh-overview
dkalinaiThats how it is setup right now more or less. My question is basically if the Real Time data chunk is just 2 hours, will it actually store it somewhere to refresh a whole days worth of data later? Because liek i mentioned i can get 2h of data at a time not more. But i obviously need it broken down by days at some point, so is accumulation going to happen still?
- mohammedadnant
Hi dkalinai
example: If today the real-time data is just 2 hours, it will work on direct query mode for today,
for tomorrow it will come under the last 3day bucket, so it will import all the data in the regular refresh.
Whether you have 2 hours or 4 hours of data, it will be treated as 1 day (date matters not the hours)
Hope this clears your doubt..
- dkalinai
I also have the possibility probably to fetch data in 2 hour chunks for previosu periods and maybe merge it in query editor? But i am not sure hwo this will work to be honest. I am mostly concerned how data is accumulating right now. For cases where a whoel day is fetched there are no issues.
- dkalinai
I came to a sad realisation that PowerBi does not save any data in it, meaning that if I am having only 1 week of data in the data source, that PowerBi will never show me more than that. Did I understand correctly? And incremental refresh only works for scheduled refreshes, where it incrementally refreshes only the last bit of data to save tiem and resources, but still the data source needs data for mroe htan a weke if you need 2 weeks of data, if each day the 7th day gets replaced then powerbi again will nto show more than 7 days right?
