Automatically append daily data that is scraped from a website to a master table

[wrwillits]

I am trying to scrape data off a public website that lists daily foriegn exchange rates, and then append that data to a master table that is capturing all historic days. 

There are systems out there that do capture all the historic foreign exchange rates but they are very expensive to use. So instead I am trying to figure out if I can create my own historic exchange rate table by just scraping each day's rates off the Reuters Currencies page (https://www.reuters.com/markets/currencies) and appending that data to a master exchange rate table in Power Query. Also, I want to set it up so that it is automatically updated daily.

Has anyone ever done this before, or is there an easier option (other than signing up for a paid service)?

[Mgreenidge]

Hi i'm trying to do something similar where I am already daily data from a ODATA connection (to Jira) and I won't the append the results as part of the automated daily refresh to another table. the Append query tabel deosn't seem  work as it's just the one table I want to "append" to.  I'd prefer to avoid a csv folder if at all possible ?

[lbendlin]

I'd prefer to avoid a csv folder if at all possible

and why would that be?

[lbendlin · Accepted solution]

You cannot do this in Power BI - it has no memory.  However, you can create a Power Automate Flow that pulls the data on a daily basis and writes it into daily CSV files. You can then point Power BI to that folder and ingest all the CSV files.  You can even implement incremental refresh based on file date but that's probably overkill given the small sizes.