Web Directory of CSV files, incremental data refresh

[Anonymous]

Hello Community,

Looking for your guidance and to learn from you.

I have a web based directory that stores CSV datasets. The datasets hold sales transactions data. A new CSV is automatically added into the directory following each day of new sales. I have attached a screengrab by way of illustrating this. It is secured with username / password, of course.

I want to build a model that can refresh / incorporate the new data as each additional CSV file is added to the directory, merging the new data with the historic data already imported into the Power BI model. I will then be able to do whatever analysis is necessary thereafter.

I only wish to reach out to the directory only to gather new data from any CSV files not prviously loaded into my Power BI model so as not to place un-necessary stress on the system.

Can you advise on the best approach to achieve this? 

As always, thank you for your kind assistance and for sharing knowledge.

Much appreciated.

[v-diye-msft · Accepted solution]

Hi @Anonymous 

Do you mean you'd like to automatically fetch new files in the directory into power bi ?

Are the files all in the same format, for example csv (comma separated values)?

Is it important that each file ends up in a separate table?

You can use Get Data - Other - Folder to ask Power Query to grab all files in a specified folder. On each refresh, all data in that folder will be read so when new files are added, they will be read into the model upon refresh. You can also use all methods available in Power Query to shape the data as you see fit. All input will however end up in one table, but with a column containing the file name so it should be a simple matter to extract the date from the file name in that column in order to allow processing of date separated data.

You might refer to this similar thread which appends new excel automatically:

https://community.powerbi.com/t5/Desktop/Get-data-from-folder-and-append/td-p/526274 

https://community.powerbi.com/t5/Desktop/How-to-dynamically-append-files-when-Folder-is-in-Microsoft/td-p/488162 

[amitchandak]

@Anonymous , refer if this can help

https://www.howtoexcel.org/power-query/how-to-extract-data-from-multiple-webpages/