Forum Discussion
Power BI incremental refresh policy limitation
Hi lbendlin
Thanks for reply.
Yes, exactly I am trying to set it up using daily partitions. Because of API data size and timeout limitations I could only set up daily refreshes. That's why I was wondering if there is some easy workaround for this because there is no option to increase days behind longer then 120 days.
From my point of view such limitation is really annoying - older data should be not erased but be still visible on reports. Older data, instead of of being deleted, should be merged into some bigger partitions like weekly or monthly and still be available since those were already transformed and recalculated. Deleting it is such a waste.
As I already wrote, in my current setup I cannot use monthly refresh because of API limitation of data size and timeouts. However even if that would be possible I my opinion setting it to monthly refresh would be very inefficient since only data from last day or two are being changed. So for example, on 30th day of some month, pulling again mostly the same, not changed data doesn't make sense. Better option would be to still have a daily refresh strategy but storing it in monthy partittions. But, as I suppose, that's not possible withing Power BI.
Please read the article again, and then set your incremental refresh to store the last three years and to refresh the last day.
- Anonymous3 years agoNot applicable
Please read my answer again.
Setting up last 3 years causes that API is being requested in 1 year chunks in the initial dataset build, which causes API faults. To prove it I could send yt video.
As I wrote in the first post, API is limited (!) to 10k records and 30secs timeout, so the question is: is it possile to achive some kind of solution with that setup which I got?
The conclusion is that API is serving small portion of data and the desired resolution would be how to catch those small chunks in dataset and build on that data analysis.
Regards,
Artur
- lbendlin3 years ago
Super User
Store these replies as Parquet files and then ingest the Parquet files into a dataset.