Passing Page values in Header in web data source and pagination

[Anonymous]

Hello Community,

I am trying to do two things here:

1. Increase the number of records in the first call to the api. default set in our application is 50. Max rows returned in one call can be 200

2. Extract the records from consecutive pages

These values cannot be passed in parameters but are within the Header. I am able to do this in Postman but not in Powerbi/query. I have tried setting Http header requests in data source settings but it's not working.

Has anyone faced this issue in the past or willing to help me?

[ImkeF · Accepted solution]

Not quite sure about what you've actually tried already, but using the Query-record should give you the chance to fill in your parameters: https://www.thebiccountant.com/2018/03/22/web-scraping-2-scrape-multiple-pages-power-bi-power-query/ 

[dax]

Hi sgoel,

As I know, there is no page_info setting in HTTP header, so I think you need to pass this in URL parameter or in M query.

I find some similar post for this problem, you could try to refer to this post and Iterating over an unknown number of pages in Power Query for details.

Best Regards,
Zoe Zhi

If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.