Forum Discussion
Retrieving paginated JSON in rate limited API
HI JeanY1,
To effectively retrieve paginated data from the Degreed API using Power Query, it's necessary to understand that Degreed uses a cursor-based pagination system, where the response includes a links.next URL pointing to the next page of data.
A common issue in implementations is failing to manage rate limits, data accumulation, and error handling properly. To comply with Degreed’s rate limit of 70 requests per minute, it’s best to introduce a delay of at least one second before each API call, ensuring you're throttling requests proactively rather than after a call.
Then, Power Query’s List.Generate function should be carefully structured to retain and combine all responses across pages, as poorly managed iterations can result in partial data loads. It's also critical to implement retry logic, particularly for handling HTTP 429 errors due to rate limits.
This involves automatically reattempting failed requests a few times before aborting, enhancing the reliability of the data pull. By restructuring the logic to fetch one page at a time with a delay, follow the links.next cursor accurately, combine results across iterations, and handle transient errors gracefully, you can build a robust Power Query solution for extracting all data from the Degreed API.
Thanks for reaching out! If this answer was helpful, please consider marking it as Accepted Solution and giving a Kudos, it helps the community!
Thank you.
Community SupportHi JeanY1,
Thank you for raising this concern. I also appreciate kushanNa for sharing a helpful reference link.
To assist you better, could you please confirm the specific API you are working with? Since APIs can vary in how they handle pagination, knowing the exact one will help us provide more accurate guidance.
Based on your description and the structure of the response (with links and data), it appears your API uses a typical RESTful pagination model. Also, kindly note that the rate-limiting logic in your current setup may not be effectively controlling the request frequency, which could be causing the stall with large datasets. Once we confirm the API and review your implementation approach, we can help refine the logic to ensure stable performance across all table sizes.
Looking forward to your response.
If this solution worked for you, kindly mark it as Accept as Solution and feel free to give a Kudos, it would be much appreciated!
Thank you.
It's Degreed. Here is the developer docs page on Pagination
I set the rate limit in the code to less than the required rate limit to make sure it wasn't an issue. What keeps happening is that multiple calls are successful but then somewhere in the middle of the calls it becomes unsuccessful.