Hello
I am a pro user pulling together a report which uses the Web connector to link to an API endpoint. This API only retreivies the newest 999 records.
I am wanting to collate the data in a Power BI model - in my mind the simplest way to do it is to have the data retreived from the API appended onto the bottom of a historic data table and remove any duplicated rows (The dataset has a unique reference column so should be easy enough) and then append new data onto this table (which has the historic data and the data from the last refresh).
The challenge I have is that I want all of those steps to happen upon refresh or when the API returns new data (I'll do a daily refresh).
Any thoughts on how this might be achieved? The issue I have with append query is that I need to keep the data from the API before it is refresh as well as the new data from the API call after the refresh.
I have tried configure the Feb 2020 incremental refresh update but I amnot confident it will work with my data sorce being the web connector.
Any help appreciated.
Thanks
