Incremental Refresh of data coming from Rest API or Json file

[walid140891]

Hello,

We are totally new to power bi and we are struggling to set the incremental refresh in order to save historical data coming from a rest API and incrementally refresh the new recent changes.

The question is: Is the incremental refresh not highly recommended for data coming from web feed (rest API) and json files due to the query folding? if yes, is there any alternative?

Thank you in advance.

[ibarrau]

Hi. I wouldn't recommend using incremental refresh with api. I don't even think you can do it. Incremental Refresh is not for every data source and must contain query folding in order to do it properly.

The best idea of having historic api data is building a script or using datafactory to populate a database, data lake or warehouse with the data. You can use Azure services to make it simple and keep in cloud. For example azure funcions calling the api and inserting the file of the day in azure data lake gen2. You can also use services like data factory requesting the api and saving data in database.

Hope this helps,

[walid140891]

Hello @ibarrau,

I really appreciate your answer, but for the moment we don't have that capabilities and time frame to apply the Azure workaround that you have mentioned, Do you have any other solution?

Thank you.

[ibarrau]

Well, you can always go ahead and do it on premise in a VM. If you don't have an azure subscription to make this easier you can write a code and run it in a local machine. Then with a Data Gateway keep Power Bi up to date.

Regards,

[walid140891 · Accepted solution]

Finally we could make the incremental refresh for data coming from rest API by following the tips in this link:

https://sqldusty.com/2020/05/20/power-bi-incremental-refresh-with-web-api-data-source/

Thank you @ibarrau .