How to import large csv to dataset using powershell

[Anonymous]

Hi,

I'm trying to load a many large csv file into powerbi automatically, and to so i've automate the creation of datasets.

However when i try to add rows of data to the dataset in this fashion it simply crashes::

`Add-PowerBIRow -DataSetId $dataset1.Id -TableName $table1.name -Rows (Import-Csv -Path ".\data.csv") ` 

as seen here :: https://docs.microsoft.com/en-us/powershell/module/microsoftpowerbimgmt.data/add-powerbirow?view=pow...

if I try iterating over every element and adding individually i get a "tooManyResquests" error, and if I try by chunks it doesn't return an error but simply nothing happens.

Is there a proper way to import a csv file into powerBI using the powershell API, in the same fashion as you would do it manually? 

thank you,

Leo

[Techinmay]

Hey @Hariharan_R 

When importing a large CSV file into a dataset using PowerShell, it is recommended to avoid loading the entire file into memory at once, as this can impact performance and lead to processing failures with very large datasets.

Instead, use PowerShell cmdlets such as Import-Csv with batch processing logic or stream data in smaller chunks for efficient handling. For exceptionally large CSV files, a better approach is to split the source file before import. In such scenario, it's recommended using SysTools CSV Splitter tool can divide a large CSV into multiple smaller files while maintaining data integrity, column headers, and record structure.

This helps improve processing efficiency, reduces memory consumption, and simplifies large-scale dataset imports in PowerShell environments. You can check it on App Store or Marketplace.

Is this helpful? Give me a thumbs up as an appreciation.

Thanks @leo

[Hariharan_R]

Hi @Anonymous 

Are you referrring the regular dataset or push datasets?

The following property is by default disabled for the dataset - "addRowsAPIEnabled": false

If you create a dataset directly using the Powershell then the above property is enabled true so you can use Add-PowerBI.

Check - https://petri.com/how-to-push-data-via-powershell-to-powerbi/

Thanks

Hari

Did I answer your question? Then please mark my post as the solution.
If I helped you, click on the Thumbs Up to give Kudos.

My Blog :: YouTube Channel :: My Linkedin