Read only new files, while keeping the old files as is

[Anonymous]

Hi,

My PowerBI app reads all files from a local folder. It has one file for everyday in the past two years (over 700 csv files). Tomorrow, when my PowerBI gets refreshed, I want it to read only that day's file (31/11/2023) instead of reading all the 700+ files which does not get updated anyway.

Although I dont want it to read all 700 files, I still am using the data in my visuals though. So I dont want it to loose the previous data, but just not re-read it every single time it refreshes,

Is this possible to do, because reading 700+ files everyday takes hours to run?

Thanks in advance!

[Data-estDog]

This is situation where you need to put your foot down and think of a more scaleable solution. 

Spin up a SQL DB for $5 a month and import the new files. Process the new files and move it to an archive folder.
Or if you are not in the cloud, SQLExpress, or MySQL. 

[Anonymous]

I do have a MySQL database.
So I can process the new files in there and append with existing data in powerBI?

[Data-estDog · Accepted solution]

You would want to import the data into your database. Append it to a central table that contains data from all the .csv files. Then Have your powerbi report point to the MySQL Table instead of the folder. 

This should be a lot faster and way less overhead (just click refresh in PowerBI or right click refresh if the report is deployed to PBI service) assuming permissions are in place.

[Data-estDog]

I should state an assumption: I assume that the row count is not the issue here. I believe the slowness is just looping through all the files.

[Anonymous]

Thanks I'll try that and see if it goes faster.

[Idrissshatila]

Hello @Anonymous , 

you could disable their refresh from power query thus they won't be refreshed but they will still be there.

[Anonymous]

But i believe this will not read the new file too right? Or I will have to read the new file and append the rows to the existing table (from previous day), which will still go on to read all the previous files wont they?

[Idrissshatila]

@Anonymous , you need to disable it to all the csv before

[Anonymous]

Well,
Sorry if I wasn't clear earlier.
But all my files are in one folder and PowerBI reads the whole folder, and stores it as one table.

[Idrissshatila]

@Anonymous , in this case you can't disable the refresh because power bi recognize it all as one table appended.

[Anonymous]

That's correct, so is there any other way?