Incremental Refresh with data for less than a day

[dkalinai]

So I know about how to enable incremental refresh in Power BI, I have it set to store 1 year and refresh last 1 day.

The issue I am having is because the data source is large it can reliably return only 2 hours of data at a time. So I can return data from 4 hours ago but upto 2 hours ago. So it means if we take current time i can not fetch last 2 hours and can only fetch the previous 2 hours.

I do not understand one thing, will this low amount of hours be able to even incrementally refreshed. Am I to expect that at the end of the day all such refreshes will add upto a day? And also the settign for live data apparently will show only last 2 hours instead.

So in a nutshell i am asking how does incremental refresh work with data which only holds a few hours in it? I have played around and reliably tested the data which arrives for entire 3 days back with this i see no issues right now.

Thanks for your help.

[dkalinai]

I came to a sad realisation that PowerBi does not save any data in it, meaning that if I am having only 1 week of data in the data source, that PowerBi will never show me more than that. Did I understand correctly? And incremental refresh only works for scheduled refreshes, where it incrementally refreshes only the last bit of data to save tiem and resources, but still the data source needs data for mroe htan a weke if you need 2 weeks of data, if each day the 7th day gets replaced then powerbi again will nto show more than 7 days right?

[dkalinai]

I also have the possibility probably to fetch data in 2 hour chunks for previosu periods and maybe merge it in query editor? But i am not sure hwo this will work to be honest. I am mostly concerned how data is accumulating right now. For cases where a whoel day is fetched there are no issues.

[mohammedadnant]

Hi @dkalinai 

This is a bit strange,

As far as I know, it all depends on the parameter that we select, in your case you can set things as per the image below, I believe you have the premium capacity to get the latest data using the direct query.

https://docs.microsoft.com/en-us/power-bi/connect-data/incremental-refresh-overview

[dkalinai]

Thats how it is setup right now more or less. My question is basically if the Real Time data chunk is just 2 hours, will it actually store it somewhere to refresh a whole days worth of data later? Because liek i mentioned i can get 2h of data at a time not more. But i obviously need it broken down by days at some point, so is accumulation going to happen still?

[mohammedadnant]

Hi @dkalinai 

example: If today the real-time data is just 2 hours, it will work on direct query mode for today,

for tomorrow it will come under the last 3day bucket, so it will import all the data in the regular refresh.

Whether you have 2 hours or 4 hours of data, it will be treated as 1 day (date matters not the hours)

Hope this clears your doubt..

[dkalinai]

So to clarify, if I am fetching data in 2 hour increments, the dashboard for the first day essentially is supposed to show only the last 2 hour increment upto the point when a whole day of data is accumulated after which i will see that day + the 2h increment. In other words in order to not see something liek that i need to set incremental refresh to not show real data? Because if we are talking about two days i might have 1 day in the view and lets say last 2 hours of the next day which will skew the visual anyways.

So in a nutshell it is iexpected from what i am hearing to see the last slice of data i refreshed only and upto the point a whole day of data is gathered?

[mohammedadnant]

Yes, right.

If you don't need the current day data then tick the box (only refresh completed days) and uncheck the box (get latest data using direct query)

[dkalinai]

Thanks, I will let the day end i guess to verify that and then if all is as you say, which i dount doubt, i will mark your answer as solution. Thanks for reassuring me.