Save scraping results with R script

[Michieldo]

Hi all,

Currently I obtain data by a R-script that scrapes data from a certain webpage. This works perfectly fine, but I would like to save the data obtained since I want to compare the results over time. Is there a way to append the newly obtained results to previous results?

Thanks!

[Greg_Deckler · Accepted solution]

Maybe. Are you doing this in an R script step? Most languages have file operations support appending to a file.

https://kb.iu.edu/d/bcrr

https://r.789695.n4.nabble.com/write-csv-with-append-TRUE-td4645712.html

[Michieldo]

@Greg_Deckler Thanks again for your help, that makes sense. I'm cleaning and combining the data in Power Query after obtaining through R. Preferably, I would like to keep the clean data and append that table. Is there a "Power BI solution" for that?

If not, I could use your suggestion and add a R-script step afterwards, write it away as a csv and import that appended csv everytime the refresh is scheduled. 

[Greg_Deckler]

Right, there is an incremental refresh option in Power BI but it only supports a limited number of data sources and there are specific requirements to get it to work. Otherwise, Power BI generally just refreshes everything and completely overwrites what is there.

https://docs.microsoft.com/en-us/power-bi/service-premium-incremental-refresh