Skip to main content
cancel
Showing results for 
Search instead for 
Did you mean: 

The ultimate Microsoft Fabric, Power BI, Azure AI & SQL learning event! Join us in Las Vegas from March 26-28, 2024. Use code MSCUST for a $100 discount. Register Now

Reply
dancarr22
Helper V
Helper V

Connecting to 'Folder' from desktop just to get info on folder contents - extremely slow

Hello,

 

We are testing Power BI to see what details it can provide about folder/subfolder/files.

So, pretty simply, pointing to a folder (in Get Data) and trying to get info about the contents of those folders.  We do not need to interrogate the contents of those underlying files.  Just need to know attributes/metadata about the file - name, folder, extension, size, last time accessed, etc.

This is extremely slow and takes several hours to run.  All we want is the metadata/properties of each file.  Is there a setting or way to do this so it runs quicker?  We do have ~40k underlying files but it took ~6 hours to run and load.  This was just one folder structure.  

 

Thanks for any help you can provided,

Dan

1 ACCEPTED SOLUTION
Greg_Deckler
Super User
Super User

Use PowerShell, this isn't really a use case Power BI was designed for. I've written multi-threaded PowerShell to scan folder structures and that's a much better tool for the job. User PowerShell to generate a CSV file of the metadata and import that into Power BI. You are trying to use a miter saw to hammer a nail.


@ me in replies or I'll lose your thread!!!
Instead of a Kudo, please vote for this idea
Become an expert!: Enterprise DNA
External Tools: MSHGQM
YouTube Channel!: Microsoft Hates Greg
Latest book!:
Mastering Power BI 2nd Edition

DAX is easy, CALCULATE makes DAX hard...

View solution in original post

2 REPLIES 2
Greg_Deckler
Super User
Super User

Use PowerShell, this isn't really a use case Power BI was designed for. I've written multi-threaded PowerShell to scan folder structures and that's a much better tool for the job. User PowerShell to generate a CSV file of the metadata and import that into Power BI. You are trying to use a miter saw to hammer a nail.


@ me in replies or I'll lose your thread!!!
Instead of a Kudo, please vote for this idea
Become an expert!: Enterprise DNA
External Tools: MSHGQM
YouTube Channel!: Microsoft Hates Greg
Latest book!:
Mastering Power BI 2nd Edition

DAX is easy, CALCULATE makes DAX hard...

@Greg_Deckler  Do you mean using PowerShell to scan Excel files and convert into metadata CSV files?

Because I have the exact  issue. Huge Excel files as data, updating monthly, and powerbi refreshes all the data sources from the beginning which took forever

Helpful resources

Announcements
Fabric Community Conference

Microsoft Fabric Community Conference

Join us at our first-ever Microsoft Fabric Community Conference, March 26-28, 2024 in Las Vegas with 100+ sessions by community experts and Microsoft engineering.

February 2024 Update Carousel

Power BI Monthly Update - February 2024

Check out the February 2024 Power BI update to learn about new features.

Fabric Career Hub

Microsoft Fabric Career Hub

Explore career paths and learn resources in Fabric.

Fabric Partner Community

Microsoft Fabric Partner Community

Engage with the Fabric engineering team, hear of product updates, business opportunities, and resources in the Fabric Partner Community.

Top Solution Authors
Top Kudoed Authors