Forum Discussion
Data Pipeline- Copy Activity - Data Source - Restful API - Pagination
If the native pagination functionality is not a feasible alternative in this case (I wouldn't know, and I don't have the option to test it as I don't have read access to an API that uses cursor), perhaps it's possible to execute API call activities within an Until loop (or a ForEach loop with the Sequential box checked).
In that case, I think you would need a string variable, you could call it varCursor and set the default value to "start".
Then use this pipeline variable varCursor to build the relative path (or query parameters, if relevant) for your API request.
Then inside each iteration of the loop, after receiving the API response, set the pipeline variable (varCursor) equal to the cursor property of the API response.
Then this updated varCursor value will be used in the API request in the next iteration.
I haven't tested it, but it sounds like something which might work.
TFE_techsupport I'm curious how to solve it in Power Query M. If you don't mind, could you please describe your solution?
I ended up building a 'until' loop with seperated copy actions and now pagination... preforms way worse but it's working.
In hindsight a self-executing powerquery function couldn've done a better job, but as i've got what i needed i'll do that next time.