Skip to main content
cancel
Showing results for 
Search instead for 
Did you mean: 

Find everything you need to get certified on Fabric—skills challenges, live sessions, exam prep, role guidance, and more. Get started

Reply
MarkCBB
Helper V
Helper V

Folder (Excel Files) vs. SQL vs. Folder CSV

Hi there,

 

I have a challange regarding the refresh time of getting data, Currently I am used Excel files in a folder (about 50 000 files, with about 30k rows each). My PC cant handle it, (64g Ram, i7 3820).

 

So my question is would it be faster to write the files, to a CSV (Many files into a CSV [Not sure how amny rows I can write to a CSV]) or upload the Excel file data into SQL or Upgrade the CPU and RAM?

 

I am stuck and testing this is taking a really long time, so I thought I would post this question in hope that I would get an answer faster than my testing.

 

Regards,

Mark B

2 REPLIES 2
v-yuezhe-msft
Microsoft Employee
Microsoft Employee

Hi @MarkCBB,

Adding to other post, if you upload Excel file data to SQL Server database, you can connect to SQL Server database using “DirectQuery” option from Power BI Desktop, then create reports in Desktop and publish them to Power BI Service. The “DirectQuery” feature lets you connect live to your data source, which should make the whole process faster as you don’t need to import data from SQL Server database to Power BI.

And after you publish the reports to Power BI Service,  Power BI automatically refreshes data in your dataset and reports hourly.

Thanks,
Lydia Zhang

Community Support Team _ Lydia Zhang
If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.

@MarkCBB My personal opinion (being a previous database guy) is to put this into SQL. Not only will it be faster, you will have the ability to backup and move the data around so much easier.


Looking for more Power BI tips, tricks & tools? Check out PowerBI.tips the site I co-own with Mike Carlo. Also, if you are near SE WI? Join our PUG Milwaukee Brew City PUG

Helpful resources

Announcements
July 2024 Power BI Update

Power BI Monthly Update - July 2024

Check out the July 2024 Power BI update to learn about new features.

July Newsletter

Fabric Community Update - July 2024

Find out what's new and trending in the Fabric Community.

Top Solution Authors
Top Kudoed Authors