Forum Discussion
Need help with loading large data volume
Hi Bobster326 ,
"Parsing the json response in power query?" - means that:
Directly call the API in Power Query using M (no need to run Python in the query step).Use Power Query functions like Json.Document to extract and transform the JSON structure into tables.Then use Power Query's merge or join capabilities to filter/match the JSON results against your CSV data using mdr_report_key.
If this post helps, then please consider Accepting as solution to help the other members find it more quickly.
Thank you.
Super UserHello Bobster326
The core issue here is hitting Power BI’s 2GB data/model limit when dealing with large API and CSV merges. Here's a short solution recommendation:
Use your Python script outside of Power BI to:
Fetch and merge the API and ZIP data.
Save the final output to a Parquet file (much smaller and optimized for Power BI).
Import the Parquet into Power BI.
This avoids memory issues, speeds up refreshes, and keeps your transformations efficient. Avoid running large async or zip-handling scripts inside Power Query—they’re not optimized for that scale
Thanks
If this solution helps, please accept it and give a kudos (Like), it would be greatly appreciated.
- lbendlin
The core issue here is hitting Power BI’s 2GB data/model limitDo you have a citation/link for that?
