Solved: Data refresh question.

TheCreepster1 · ‎09-14-2017

Hi all, I have a large dataset that grows daily. I'm looking to use power bi as a way of creating some visuals for this data to make it easier to understand (guess that's what we're all doing really!)

Anyway, my question is about the data refresh. As I mentioned, my dataset is massive, and what I've noticed when refreshing smaller datasets is that it brings the data in as new each time, instead of just bringing in the newest data. I have the option of pulling data in from SAP HANA, which is where the data table is, or exporting information daily to a .xlsx file in a location I can then point Power BI at. Is there anyway to just bring in the latest data and append it, rather than have to refresh the entire dataset again?

Anonymous · ‎09-14-2017

In short, if you are using "Import" the answer is no. It is always a case that Power BI clears and fetches the data again. This means you need to make sure your import statements are contrained to only get the data you require. For example exclude rows and columns that you don't need.

As an alternate, if your data source is truely massive and you don't need every single function, you could consider a Direct Query. This will reduce how much you download in a single hit, but will require a more sustained performance hit on your data bases.

View solution in original post

TheCreepster1 · ‎09-18-2017

Thank you, the data set is quite wide and about 300 million records in total. We'll look at what we can do to limit the data that comes in and have a go at setting up a direct query with parameters. Thanks for your response!

Anonymous · ‎09-14-2017

In short, if you are using "Import" the answer is no. It is always a case that Power BI clears and fetches the data again. This means you need to make sure your import statements are contrained to only get the data you require. For example exclude rows and columns that you don't need.

As an alternate, if your data source is truely massive and you don't need every single function, you could consider a Direct Query. This will reduce how much you download in a single hit, but will require a more sustained performance hit on your data bases.