Forum Discussion

ismael_spadetto's avatar
ismael_spadetto
Frequent Visitor
3 years ago

Incremental Refresh for Dataflow calling Source API Multiple Times

We have Dataflows that utilise API Sources that we would like to refresh incrementally and the incremental refresh is not working as expected. The issue we are seeing is that each time the increme...
Dataflow
cpwebb's avatar
cpwebb
Microsoft Employee
3 years ago

I can see that in the screenshots showing the API activity there is a parameter of &limit=10001 attached to the API requests. This suggests Power Query is calling the API to try to get the schema of the response. Can you try turning off all data privacy settings for the dataflow? You can do this by going to the Options dialog in Power Query Online, going to Project/Privacy and selecting "Allow combining data from multiple sources" - this may help.

  • OllieSvT's avatar
    OllieSvT
    Regular Visitor
    2 years ago

    Hi cpwebb, I have been having the exact same issue as described by ismael. 

    I checked my PQ online and can confirm the same thing happens even with the 'allow combioning...' option selected. I am connecting into a freely available database with an API. This has a fair usage limit on it which I am currently breaching due to the way PBI is pulling the data when using an incremental refresh. I have a temporary solution in place that works outside of PBI but I wish to develop an all encompassing dataflow for the long term and this process will be key in the long term stability of this. 

     

    Any thoughts on how to improve the efficiency of the API/Incremental Refresh process is much appreciated.
    Thanks
    Ollie