API Pagination with scroll ID

Yes, you’re on the right track. The behavior you’re seeing—very high row counts with mostly duplicates, usually happens when the Scroll ID is not being passed correctly in subsequent requests. In scroll-based pagination, each response returns a new _scroll_id, and that value must be used in the next call to fetch the next batch of records.

In your query, the main issue looks to be how the parameter is passed in the  function. The API returns _scroll_id, but for the next request it should typically be sent as GetScroll (without the underscore and with correct casing). If this is not handled properly, the API may keep returning the same page repeatedly, which explains the duplicates.

Also, make sure your Authorization header includes the Bearer prefix, as some APIs require that for proper authentication.

Once the scroll parameter is corrected and the new Scroll ID is passed on each iteration, the results should return unique pages and align more closely with the expected total row count.

Thanks,

Prashanth