Forum Discussion
How to import large csv file in powerBI
I recently dealt with a similar issue while importing a very large CSV into a dataset using PowerShell. For standard files, you can use Import-Csv and process records in batches, but with huge CSV files, PowerShell may become slow or consume too much memory. A practical approach is to import data in chunks instead of loading the entire file at once. If the CSV is extremely large, splitting it beforehand makes the process much easier and improves performance. For that, a CSV Splitter tool can help divide one large CSV into multiple smaller files while preserving rows, headers, and data structure. It makes large dataset imports more manageable and reduces the chances of failures or performance issues during PowerShell processing.
4 Replies
- oussamahaimoud
Memorable Member
Hi Techinmay,
Here's a robust, production-grade approach for handling very large CSV imports. For truly massive datasets, skip the Push API and use Fabric's native ingestion:
# Upload split CSVs to Fabric Lakehouse via Azure Data Lake / OneLake
$storageUrl = "https://onelake.dfs.fabric.microsoft.com"
$workspace = "your-workspace-name"
$lakehouse = "your-lakehouse-name"
Get-ChildItem "C:\Data\Split\batch_*.csv" | ForEach-Object {
$destination = "$storageUrl/$workspace/$lakehouse.Lakehouse/Files/imports/$($_.Name)"
Invoke-RestMethod -Uri $destination -Method PUT `
-Headers @{ Authorization = "Bearer $token" } `
-InFile $_.FullName
Write-Host "Uploaded: $($_.Name)"
}
Then trigger a Fabric Notebook or Data Pipeline to load from the Files section into a Delta table.
Remember that :
Push API row limit: 10,000 rows per call, 1M rows/hour per dataset
Fabric OneLake: No practical file size limit
Dataflow Gen2: Best for transformation + load at scale
Always use UTF-8 encoding to avoid character corruption across splits
Hope this helps! Don't forget to mark as solution and thumbs up, that's motivate me to keep helping 🙂
- Olufemi7
Super User
Hello Techinmay,
You do not really need PowerShell for this. Microsoft Power BI Desktop can handle large CSV imports directly.
What normally helps is:
1. Importing through Transform Data instead of loading straight away
2. Removing unused columns early
3. Filtering rows before doing other transformations
4. Setting data types manually if detection becomes slow
If the data is split across multiple CSVs, the Folder connector works well too.
CSV sources also do not support query folding, so all transformations happen locally in Power Query. Keeping the dataset as small as possible early in the steps usually improves performance.Hey Everyone,
I searched a lot for how I could easily import a large CSV file into Power BI, then I found out that if your CSV file is too large to import into Microsoft Power BI, the best solution is to split the CSV into smaller parts before importing.
As it can cause slow loading, memory issues, or import failures in Power BI. So for that, I found one of reliable software, which is SysTools CSV Splitter Tool to divide a huge CSV file into multiple smaller files without losing any data. The software allows users to split CSV files by rows, size, or columns, which makes it easier to manage and upload data into Power BI smoothly.
After splitting the file, simply import the smaller CSV parts one by one into Power BI for faster processing and better performance.
If someone is also facing the same issue, then they can refer to this solution.
Thanks,
Techinmay
v-veshwara-msftCommunity Support
Hi Techinmay ,
Thanks for posting in Microsoft Fabric Community and for sharing your approach in importing large csv files into Power BI.Also, thank you to the community members for sharing their valuable suggestions and approaches around handling large CSV imports in Power BI/Fabric. The recommendations provided cover different scenarios and should help others facing similar challenges.
Please let us know if you have any additional questions or if you need further clarification on your specific use case. Feel free to reach out and we would be happy to assist.
Thank you.
