Forum Discussion
Need help with loading large data volume
Hi Bobster326 ,
"Parsing the json response in power query?" - means that:
Directly call the API in Power Query using M (no need to run Python in the query step).Use Power Query functions like Json.Document to extract and transform the JSON structure into tables.Then use Power Query's merge or join capabilities to filter/match the JSON results against your CSV data using mdr_report_key.
If this post helps, then please consider Accepting as solution to help the other members find it more quickly.
Thank you.
Community SupportHi Bobster326 ,
Thank you for reaching out to Microsoft Fabric Community.
Thank you lbendlin for the prompt response.
In addition to Ibendlin's points:
As you are running into Power BI’s file and memory limitations, especially when dealing with large datasets (>2GB). There are a couple of options as suggested by Ibendlin:
parsing JSON directly in Power Query or storing the data in a SQL database are both solid options.
However, given the 2GB+ data size, I would suggest:
1.Using Python externally to fetch/merge the data
2.Saving it to a SQL database or a parquet file
3.Then loading into Power BI - this avoids memory and refresh limits, while keeping performance smooth.
Parsing JSON in Power Query works but may be slower and hit memory issues if the dataset is large or heavily paginated.
If this post helps, then please consider Accepting as solution to help the other members find it more quickly.
Thank you.