Guidence Needed, best way to Get Data from Large CSV files in a Folder updated into Premium per user

[nrowey]

I have Large 15 -20 GB CSV files in Folders on Sharepoint , When I try to refresh and combine and these large .csv's I get Run Out of Mem error in a DataFlow.
Is a Data Flow the best way to approach this and will I be able to do an Incremental Refresh on Files in a Sharepoint folder , I can get a date field with a bit of manipulation. Do I need to Purchase more Storage for this from MS ? if so what kind and how..

Only the first few files in each folder are this large future files will be 5 to 20 MB.

Thanks for help...

[christinepayton · Accepted solution]

Your best bet is to load the CSVs into a real database like SQL and connect to that. I don't think you will get good performance with 30GB of CSVs in SharePoint no matter what you do (if it'd work at all).