Skip to main content
cancel
Showing results for 
Search instead for 
Did you mean: 

Get certified in Microsoft Fabric—for free! For a limited time, get a free DP-600 exam voucher to use by the end of 2024. Register now

Reply
joncmaloney
New Member

Incremental Refresh Using M Query and paginated API

I'm trying to use an M Query to pull data from a paginated API. Currently the M Query works to pull all the paginated API data using
Json.Document(Web.Contents(Url))
The above line is run through a loop until there are no results returned.

When I try and use Incremental Refresh the service fails to refresh the data source. Is there a way to use Incremental Refresh to obtain Paginated API data via an M Query?

7 REPLIES 7
lbendlin
Super User
Super User

Incremental Refresh is date time based. You need to have a filter transform in Power Query that includes the RangeStart and RangeEnd parameters.  So your API call must allow you to limit date time ranges. Then you can hand over these parameters to the API call, do your pagination, and return the combined data for that date time range. 

 

NOTE:  This is INCREMENTAL refresh, not DIFFERENTIAL refresh. It does not handle data changes well.

Hi @lbendlin

 

Thanks for the reply. To be more specific when I have added RangeStart and RangeEnd based on this guide: 

https://docs.microsoft.com/en-us/power-bi/connect-data/incremental-refresh-configure I append RangeStart and RangeEnd to the URL string that is used in: Json.Document(Web.Contents(Url)) as a query param. I can change the RangeStart and RangeEnd parameters reload the data and the dataset loaded into Power BI desktop shows new data based on the updated parameters. After I have setup the M Query I then publish the Power BI Query to a service (I believe incremental refresh is only possible when using a service). However when I have a look at the service I see the following error message about scheduled refresh. 

 

joncmaloney_0-1653353390877.png

Does this error message mean that the Url i'm using cannot be reached by the service or does it mean that M Queries cannot be used with scheduled refresh?

 

Thanks again for your assistance so far with this issue. 

 

Kind Regards, 

Jon

where does the link go?

Link goes to the server with the API. However when we are using the destop version the pc would route to the API differently compaired to a service hosted in the cloud. So i'm guessing what your saying is that yes the service can't see the API and that's why I'm getting the error? I just need to fix connectivity?

Sorry, I meant where the "Discover Data Sources"  link goes.

The link goes to https://app.powerbi.com/. It wasn't hepful at all. 

Without access to your API I can't help more (and you may not want to give that access) . If you have a Pro license you can consider raising a Pro ticket at https://powerbi.microsoft.com/en-us/support/pro/

Helpful resources

Announcements
November Carousel

Fabric Community Update - November 2024

Find out what's new and trending in the Fabric Community.

Live Sessions with Fabric DB

Be one of the first to start using Fabric Databases

Starting December 3, join live sessions with database experts and the Fabric product team to learn just how easy it is to get started.

Las Vegas 2025

Join us at the Microsoft Fabric Community Conference

March 31 - April 2, 2025, in Las Vegas, Nevada. Use code MSCUST for a $150 discount! Early Bird pricing ends December 9th.

Nov PBI Update Carousel

Power BI Monthly Update - November 2024

Check out the November 2024 Power BI update to learn about new features.