Forum Discussion
Power BI incremental refresh policy limitation
Super User
Power BI policy has a hard limit to 120 days
I am not aware of such a limit. Did you mean 120 partitions? But even that is not a limit mentioned anywhere. Maybe you are trying to store daily partitions? If so then change that to monthly, quarterly or yearly.
For example - you can choose to store the last 10 years and refresh the last one day. Power BI service will automatically maintain and consolidate the required partitions.
I think your bigger issue is the passive-aggressive behavior of the API. You may want to consider a different process to fetch the data (via Power Automate, or PowerShell) to make sure you don't miss any data.
Hi lbendlin
Thanks for reply.
Yes, exactly I am trying to set it up using daily partitions. Because of API data size and timeout limitations I could only set up daily refreshes. That's why I was wondering if there is some easy workaround for this because there is no option to increase days behind longer then 120 days.
From my point of view such limitation is really annoying - older data should be not erased but be still visible on reports. Older data, instead of of being deleted, should be merged into some bigger partitions like weekly or monthly and still be available since those were already transformed and recalculated. Deleting it is such a waste.
As I already wrote, in my current setup I cannot use monthly refresh because of API limitation of data size and timeouts. However even if that would be possible I my opinion setting it to monthly refresh would be very inefficient since only data from last day or two are being changed. So for example, on 30th day of some month, pulling again mostly the same, not changed data doesn't make sense. Better option would be to still have a daily refresh strategy but storing it in monthy partittions. But, as I suppose, that's not possible withing Power BI.
- lbendlin
Please read the article again, and then set your incremental refresh to store the last three years and to refresh the last day.
Please read my answer again.
Setting up last 3 years causes that API is being requested in 1 year chunks in the initial dataset build, which causes API faults. To prove it I could send yt video.
As I wrote in the first post, API is limited (!) to 10k records and 30secs timeout, so the question is: is it possile to achive some kind of solution with that setup which I got?
The conclusion is that API is serving small portion of data and the desired resolution would be how to catch those small chunks in dataset and build on that data analysis.
Regards,
Artur
- lbendlin
Store these replies as Parquet files and then ingest the Parquet files into a dataset.