Forum Discussion
Trigger full Refresh on Dataset with Incremental Refresh
I need a way to perform a "full" refresh of my 10 days window once a day with incremental refreshes in between.
Background
I have a very large datasource with billion of records in the future and past.
I only need a current sub-set of these records at a time. But even this sub-set is very large (approx. 500mio recrods each).
The reports need to be very up-to-date, but direct query is not an option for legal reasons.
Most of the data (98%) does not change. So a full refresh at this order of magnitute every 5min or so does not seem like the way to go.
Incremental refresh appears suitable.
Problem is my rolling window of only 10 current days.
Much of the data will not have changed since the last incremental refresh, thus not being pulled once the windows shifts one day.
I would need a way to perform a "full" refresh of my 10 days window once a day with incremental refreshes after that.
Anyone knows of a way to do this prrgamatically?
Many thanks for your input!
Jan
3 Replies
Hey JSiebrecht ,
my 1st idea is to an incremental refresh by using the xmla endpoint, but this requires some kind of Premium capacity: Advanced incremental refresh with the XMLA endpoint in Power BI - Power BI | Microsoft Docs
Another solution is to create a view on top of your data source, that filters the required data in advance. Then you can publish your dataset each day (or whenever you want) and perform an initial load, after that you can use the incremental load.
Hopefully, this requires some ideas on how to tackle your challenge.
Regards,
Tom
Hi TomMartens ,
thanks for your input!
My dataset is hosted in a P2 SKU Premium Capacity. If this gives options.
To publish the dataset each day for the next X years, I don't hink that's a viable option for this kind of business process.
Wondering if you have found a solution to this issue
