Folder (Excel Files) vs. SQL vs. Folder CSV

[MarkCBB]

Hi there,

I have a challange regarding the refresh time of getting data, Currently I am used Excel files in a folder (about 50 000 files, with about 30k rows each). My PC cant handle it, (64g Ram, i7 3820).

So my question is would it be faster to write the files, to a CSV (Many files into a CSV [Not sure how amny rows I can write to a CSV]) or upload the Excel file data into SQL or Upgrade the CPU and RAM?

I am stuck and testing this is taking a really long time, so I thought I would post this question in hope that I would get an answer faster than my testing.

Regards,

Mark B

[Anonymous]

Hi @MarkCBB,

Adding to other post, if you upload Excel file data to SQL Server database, you can connect to SQL Server database using “DirectQuery” option from Power BI Desktop, then create reports in Desktop and publish them to Power BI Service. The “DirectQuery” feature lets you connect live to your data source, which should make the whole process faster as you don’t need to import data from SQL Server database to Power BI.

And after you publish the reports to Power BI Service,  Power BI automatically refreshes data in your dataset and reports hourly.

Thanks,
Lydia Zhang

[Anonymous]

@MarkCBB My personal opinion (being a previous database guy) is to put this into SQL. Not only will it be faster, you will have the ability to backup and move the data around so much easier.