Forum Discussion
Extract data from a URL with pagination, when the cursor is randomized
Hi,
I am trying to extract information from export links, where the data is paginated and the cursors are a random combination of numbers, letters and signs.
I find a lot of solutions for extracting data when the cursor is numerical and logically numbered (for example IMDB-examples where you enter pagination=1, 51, 101 and so on). Does anyone know how to work around this when there is no logic in the cursor-values?
The data I'm trying to extract is built up like this: https://www.########.net/Flex/1145365367436.json?t=AKHJBKHJKjhvykjbøpKKIUG-uujhvyiyu-jyyUYVUY&pagination=1.
When retrieving the data from the first page in pagination, the cursor value for the next page is in the meta. To extract the next page in pagination the URL logic is like this https://www.########.net/Flex/1145365367436.json?t=AKHJBKHJKjhvykjbøpKKIUG-uujhvyiyu-jyyUYVUY&pagination=1&cursor=*cursor value*
and so on.
Very grateful for any help!
1 Reply
Use List.Generate or List.Accumulate. Or use a recursive function that exits when there's no more "next page" link.
It is very hard/close to impossible to help without access to the API (which you understandably shouldn't grant us).
This link shows a boilerplate implementation. Handling paging for Power Query connectors - Power Query | Microsoft Learn
