Updating a Combined and Transformed Dataset

[Bebo]

I am trying to determine the best method to conduct daily updates of a previously Combined and Transformed dataset.

My source data is a ahred drive folder which contains CSV files )one file for each day). Each day, a new CSV file is sdded to this folder to reflect activity of that day. I have figured out how to Combine and Transform the files in this folder on a one time basis. The transfomation being performed is adding a couple of columns to the base data. My question is how can I do a daily update of this Combined and Transformed dataset (add new daily CSV file to the dataset)?

[Icey]

Hi @Bebo ,

---

My source data is a ahred drive folder which contains CSV files )one file for each day). Each day, a new CSV file is sdded to this folder to reflect activity of that day.

The transfomation being performed is adding a couple of columns to the base data. 

---

Do you mean that you want to add a new column for each day's new CSV file? Could you give us some sample to show your data structure? Please don't contain any sensitive information.

Reference: How to provide sample data in the Power BI Forum - Microsoft Power BI Community

Best Regards,

Icey

[lbendlin]

Adding columns?  That is not really something you want to do on a daily basis.  Combining (appending) the CSV files to a master table is more common.

In Power Query you can connect to a folder rather than a file. Then you can collect all the relevant CSV files in that folder and have the Power Query script extract the file contents and combine it.  If you want to you can also implement incremental refresh based on the file creation timestamps.