How to model queries in Power BI for daily append of new data instead of overwriting

[Anonymous]

Hi,

I'm trying to build a simple report based upon data published on a website. Here is what I want to achieve

1. This website publishes data for COVID cases in the country.
2. The number are just the current numbers, without any time-series.
3. I want to fetch these numbers from this website daily and build a report on top of it (with time series kind of analysis).
4. So I fetch these numbers (Get Data > Web > URL) and get this into a query
5. I then add a custom column with a timestmap (M's DateTime.LocalNow() fucntion) and get this data with the required timestamp.
6. Now I want to refresh this query daily, so that I get daily results in this query.
7. As expected, PBI simply overwritse the existing rows with new data, with the latest timestamp (my custom column).

I tried few things like:-

1. Creating a new query and appending data to it, it doesn't seem to work, existing data gets over-written (maybe the way I have created the new query).
2.  Explored incremental refresh, it doesn't seem to fit my use case.
3. Tried looking at other similar posts, none seem to help me resolve this.

Questions:-

1. Is there a simple workaround to circumvent this (point#7) and have PBI append new data instead of overwriting existing data.
2. Am i correct on point#2 above (incremental refresh)? 

Appreciate any pointers. Thanks in advance!

[FrankAT]

Hi @Anonymous ,

which URL do you use to get the data?

Regards FrankAT

[Greg_Deckler]

Couple options. 

One, create a new query. Disable refresh on the original query. Create an Append query as New that appends the new query to the original query.

Or, download the files to a folder every day. Use a Folder query.

[Anonymous]

Option#1: Based upon what I could try, I could see my append query's data was also got overwritten during refresh, which I didn't want. Is it really an option? May be I didn't implement it the way it shoud have.

Option#2: I did some further lookup and found that this is probably the right way to do this, have my own little staging and transformation layer (add time series attribute) and then have PBI connect to this layer.

Thanks.