Forum Discussion

rafael-b's avatar
rafael-b
Frequent Visitor
5 years ago

Import from local vs import from BigQuery

Hi all,

I'm currently working with data that comes in form of one CSV file per month, containing over 15mn rows each. I'm required to have at data going back to at least January 2018, so that's about 400mn rows now. I have compressed each month's file into parquet, kept them in my local disk, and then I import each one to PBI, where I append them all to make visuals.

The load process to PBI is currently very slow and I'm looking into incremental refresh for next month's data. Maybe you already have some suggestions regarding the steps I currently go through, but that's not the main question here.

 

As I said, I'm keeping the files on my disk and I don't think that I'll be able to do this forever. The other option I see is uploading the tables to BigQuery (my company uses Google Cloud Platform already) and importing them from there (not Live Query). I'm wondering if the import and refresh from BigQuery will be much slower then importing from the disk. If so, would you have suggestions regarding the file storage and the imports to PBI?

 

Thanks!!

1 Reply