Question Regarding Data Refresh Frequency and History

[Alkhos]

Hello,

I have two questions regarding data refresh in PBI.

I am using PBI pro and enterprise gateway to referesh data from a SQL server, what I want to know about it are:

1) I see that the refresh frequency has options of "daily" and "weekly". Is there any option to refresh with a higher frequency, e.g. every 15 minutes?

2) When a refresh is happening, does my older data get overwritten? We keep data for the past two days in our database, but ideally, I would like the data that the gateway fetches when refreshing to be appended to whatever data I have so that my report can use data over a longer timespan. e.g. if I run the gateway for few months, I prefer to have all the data for, say, the last month that was eventually fetched from the db over the past rather than having the last two days. Is such a capabiliy supported in PBI? and if it is, do I need any settings in my gateway? If it is not, is there any alternative or workaround to keep longer span of data and keep appending to it and with every refresh, only retire the oldest set of data over the timespan?

Thanks

Ali

[Greg_Deckler · Accepted solution]

Pro supports Hourly refresh, pick Daily, pick an hour then create another refresh, pick Daily, pick another hour, etc. If you need more frequently you should investigate Direct Query or a streaming data source. Using these methods, you can get "live" data (up to the second)

Currently, refresh in Power BI is always a complete refresh of the data. So, your situation, you would need to keep the other data around somewhere and append your current data. You could do this with an R script Query and an Azure SQL database for example. You would create a query that grabbed everything from your Azure SQL database. You would then create another query that grabbed from the data source you mention that only holds 2 days of information. You would do this in R, for example, and this query would write back to the Azure SQL database with the new information. You would then Append this query to your first query. In this way, you would get everything. I believe there is an Idea for incremental refresh. Ah yes, here it is, you should vote for it:

https://ideas.powerbi.com/forums/265200-power-bi-ideas/suggestions/7288623-incremental-data-loads