Retrieving paginated JSON in rate limited API

HI JeanY1,

To effectively retrieve paginated data from the Degreed API using Power Query, it's necessary to understand that Degreed uses a cursor-based pagination system, where the response includes a links.next URL pointing to the next page of data.

A common issue in implementations is failing to manage rate limits, data accumulation, and error handling properly. To comply with Degreed’s rate limit of 70 requests per minute, it’s best to introduce a delay of at least one second before each API call, ensuring you're throttling requests proactively rather than after a call.

Then, Power Query’s List.Generate function should be carefully structured to retain and combine all responses across pages, as poorly managed iterations can result in partial data loads. It's also critical to implement retry logic, particularly for handling HTTP 429 errors due to rate limits.

This involves automatically reattempting failed requests a few times before aborting, enhancing the reliability of the data pull. By restructuring the logic to fetch one page at a time with a delay, follow the links.next cursor accurately, combine results across iterations, and handle transient errors gracefully, you can build a robust Power Query solution for extracting all data from the Degreed API.

Thanks for reaching out! If this answer was helpful, please consider marking it as Accepted Solution and giving a Kudos, it helps the community!
Thank you.