Skip to main content
cancel
Showing results for 
Search instead for 
Did you mean: 

Get certified in Microsoft Fabric—for free! For a limited time, the Microsoft Fabric Community team will be offering free DP-600 exam vouchers. Prepare now

Reply
hly
Frequent Visitor

How to query new random (dated) incoming data files from azure blob storage account into Power BI?

How to query new random (dated) incoming data files from azure blob storage account into Power BI?

 

For example, DataFile20240904.csv, DataFile20240927.csv, DataFile20250212, and random future dated files. The columns and format are exactly the same, it is just new data that's flowing in.

 

The goal is to ensure that when the refresh schedule is set up daily, that it will automatically pick up these files, and 'APPEND' into a single master table in Power BI (e.g. DataFileALL).

 

Does anyone know how this can be implemented?

3 REPLIES 3
lbendlin
Super User
Super User

The usual approach is to use a folder type connector, then filter for the files you want (by location or extension, for example)  and then use Table.AddColumn to consume the contents of each file and then combine them into the final table.

 

Forget about the "APPEND" part. That's not something Power BI can do, but as long as your files are CSV or Parquet it doesn't matter.

hly
Frequent Visitor

With the 'folder connector', will this also work with a folder within Azure Blob Storage Account?

 

As for 'APPEND', I use this feature in Power BI/Query all the time. It works well when the filename is known. In the case of random future-dated files...I don't see a solution for it.

hly_0-1725336861878.png

 

Hi,

Thanks for the solution lbendlin provided, and i want to offer some more information for user to refer to.

hello @hly , based on your description, you can use the connector Azure Blob storage, it can include the files in your connector, then you can use transform multiple files in folder in power query, you can refer to the following link.

Import Multiple Files from a Folder with Power Query - BI Gorilla

Get data from folder in Power Query: combine files quickly (exceloffthegrid.com)

 

Best Regards!

Yolo Zhu

If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.

 

Helpful resources

Announcements
OCT PBI Update Carousel

Power BI Monthly Update - October 2024

Check out the October 2024 Power BI update to learn about new features.

September Hackathon Carousel

Microsoft Fabric & AI Learning Hackathon

Learn from experts, get hands-on experience, and win awesome prizes.

October NL Carousel

Fabric Community Update - October 2024

Find out what's new and trending in the Fabric Community.

Top Solution Authors