How could I append rows to the table created through custom connector?

[cte_crest]

Hi,

I am new to Power Bi and currently, I am trying to implement a solution using custom connectors. 

In this custom connector, I bring the data through rest APIs. So, I think it won't have any query folding support. The source from which I bring the data has many rows (100s of thousands daily), and I want to bring that data to Power BI. Now, is there any way that we can only fetch the latest data only? 

For example, if I have collected the data from 1st Jan 2020 to 31st March 2020. Then, is there any way that when I do the next refresh, I only bring the data from 31st March 2020 to today's date, let's say 7th April 2020, and append that data to the data that I have previously fetched?

While going through the Github connector, I found that the data is fetched again every time and is re-written to the table. So, is there a way to preserve the data which is already fetched and only add the new data pulled from the API?

[Greg_Deckler · Accepted solution]

Not generally, incremental refresh is only available for a certain few types of data sources

[cte_crest]

@Greg_Deckler 

In this case, it would be great if you help me determine what would be a good way to implement this particular use case? I tried with the API of powerbi, but it has limit of maximum number of rows (5 million).

Again, thanks for the quick reply.