Incremental Refresh Issues

Anonymous · ‎04-15-2020

Hello

I am a pro user pulling together a report which uses the Web connector to link to an API endpoint. This API only retreivies the newest 999 records.

I am wanting to collate the data in a Power BI model - in my mind the simplest way to do it is to have the data retreived from the API appended onto the bottom of a historic data table and remove any duplicated rows (The dataset has a unique reference column so should be easy enough) and then append new data onto this table (which has the historic data and the data from the last refresh).

The challenge I have is that I want all of those steps to happen upon refresh or when the API returns new data (I'll do a daily refresh).

Any thoughts on how this might be achieved? The issue I have with append query is that I need to keep the data from the API before it is refresh as well as the new data from the API call after the refresh.

I have tried configure the Feb 2020 incremental refresh update but I amnot confident it will work with my data sorce being the web connector.

Any help appreciated.

Thanks

mwegener · ‎04-15-2020

Hi @Anonymous

look at this concept.

https://www.thebiccountant.com/2020/04/13/create-a-load-history-or-stage-in-cds-instead-of-increment...

Did I answer your question?
Please mark my post as solution, this will also help others.
Please give Kudos for support.

Marcus Wegener works as Full Stack Power BI Engineer at BI or DIE.
His mission is clear: "Get the most out of data, with Power BI."
twitter - LinkedIn - YouTube - website - podcast - Power BI Tutorials