Large csv data file of about 30gb

[blesd]

I have created a Power BI report using a sub-set of a large dataset. I would like to update the data source with a csv data file of about 30gb

But it's taking too long and the remote desktop keeps breaking off, please is there a walk around this.

I would really appreciate your response, the solution is required urgently.

[blesd · Accepted solution]

Thanks.

These are the steps I took to make it work
1. I changed the following data load options settings:
     Unchecked auto date/time
     Disabled parallel loading of tables
     Unchecked the background data option 

2. I also increased the data management cache option and used a system with a higher ram.

[v-yueyunzh-msft]

Hi , @blesd 

 If you are a Power BI free or pro user, when you publish a PBIX file over 1GB to Power BI Service, you will get a file size limit message, but there is no limit about file size in Power BI Desktop, you should be able to load data from CSV files to Power BI Desktop.

In addition, you can load data from CSV files to SQL Server table, and you can use DirectQuery mode to connect to the SQL Server source in Power BI Desktop when you have large amount data in the table.

For more information, you can refer to :

Import csv into SQL server (with query OR without query using SSMS) - QA With Experts

Using DirectQuery in Power BI - Power BI | Microsoft Learn

Best Regards,

Aniya Zhang

If this post helps, then please consider Accept it as the solution to help the other members find it more quickly

[aj1973]

Hi @blesd 

With a pro license you can only have a dataset up to 1G of size. Not sure what could the impact of a your CSV file on the size of the Dataset when you import it...if it is more than 1G then the desktop will time out.

Best solution would be to use the Direct Query mode connection to your CSV file.