How To Setup Power Bi refresh for large dataset with rest API.

[dsvansh]

Hello everyone,

I’m working on a Power BI report that retrieves data from REST APIs. The problem is that the dataset has grown too large, and the API server can’t handle returning all records during refresh, even with pagination enabled.

I considered using Incremental Refresh, but it still requires a full initial refresh after publishing to create the partitions — and that initial load fails due to data volume limitations.

I want to know if there is any approach where Power BI can append only the newly updated daily data, without requiring a full refresh of all historical data every time.

Thanks in advance.

[v-pgoloju]

Hi @dsvansh,

Just following up to see if the Response provided by community members were helpful in addressing the issue. if the issue still persists Feel free to reach out if you need any further clarification or assistance.

Best regards,
Prasanna Kumar

[v-pgoloju]

Hi @dsvansh,

Thank you for reaching out to the Microsoft Fabric Forum Community, and special thanks to @GilbertQ and @lbendlin  for prompt and helpful responses.

Just following up to see if the Response provided by community members were helpful in addressing the issue. if the issue still persists Feel free to reach out if you need any further clarification or assistance.

Best regards,
Prasanna Kumar

[GilbertQ]

Hi @dsvansh 

A way you could extract the data to Sharepoint is to use Power Automate and query the API install the information as recommended by @lbendlin  

[lbendlin · Accepted solution]

Extract the REST API data into JSON, CSV, or Parquet files as needed  (including re-fetching if changes happened).  Store the result in OneLake or another suitable file store (SharePoint works in a pinch)

Ingest all the data, or access it via Direct Lake, or construct your own incremental refresh - up to personal preferences

If the partitions grow too big, use (fake) incremental refresh with bootstrapping and XMLA refresh processes.

Troubleshoot incremental refresh and real-time data - Power BI | Microsoft Learn