Forum Discussion
How to query new random (dated) incoming data files from azure blob storage account into Power BI?
How to query new random (dated) incoming data files from azure blob storage account into Power BI?
For example, DataFile20240904.csv, DataFile20240927.csv, DataFile20250212, and random future dated files. The columns and format are exactly the same, it is just new data that's flowing in.
The goal is to ensure that when the refresh schedule is set up daily, that it will automatically pick up these files, and 'APPEND' into a single master table in Power BI (e.g. DataFileALL).
Does anyone know how this can be implemented?
3 Replies
The usual approach is to use a folder type connector, then filter for the files you want (by location or extension, for example) and then use Table.AddColumn to consume the contents of each file and then combine them into the final table.
Forget about the "APPEND" part. That's not something Power BI can do, but as long as your files are CSV or Parquet it doesn't matter.
With the 'folder connector', will this also work with a folder within Azure Blob Storage Account?
As for 'APPEND', I use this feature in Power BI/Query all the time. It works well when the filename is known. In the case of random future-dated files...I don't see a solution for it.
Hi,
Thanks for the solution lbendlin provided, and i want to offer some more information for user to refer to.
hello hly , based on your description, you can use the connector Azure Blob storage, it can include the files in your connector, then you can use transform multiple files in folder in power query, you can refer to the following link.
Import Multiple Files from a Folder with Power Query - BI Gorilla
Get data from folder in Power Query: combine files quickly (exceloffthegrid.com)
Best Regards!
Yolo Zhu
If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.
