Fabric Data Factory Copy Activity Pagination Rules with QueryParameters receive 400 BadRequest error

[fabric-dev79]

Good day Fabric Community,

I need some help. Trying to implement pagination rules for a Copy Activity using a REST Api source in Fabric Data Factory.

I have used the following links (combined with my general understanding of using Data Factory for many years) to verify my understanding but unfortunately my exact scenario is not clearly explained hence why I am reaching out to the community for some help.

https://learn.microsoft.com/en-us/azure/data-factory/connector-rest?tabs=data-factory#pagination-sup...

https://techcommunity.microsoft.com/blog/fasttrackforazureblog/implementing-pagination-with-the-copy...

The scenario:

I have an API that return multiple pages. Each page will include a "cursor" that points to the next page of results as follow

    "cursor": {

        "next": "XXXXXXXX"

    }

When there are no more pages to process, final page will have the next value set to null:

    "cursor": {

        "next": null

    }

The configuration of my copy activity source is as follow:

Relative URL set to  https://my.api.com/transactions?cursor={cursor_var}.   * Note: I have used a dummy URL for illustration purposes

Pagination Rules:

I have also tried a different permutation:

I have verified the API call works by way of debugging the API via a Web Activity and can confirm that a value is returned for cursor.next (verified by reading the value and successfully writing it to a pipeline variable) and therefore comfortable that the API endpoint and parameters that the API support works as expected. I have also verified this by replicating the API GET using Postman. That all checks out ok.

When trying the Pagination rule configuration above I get the following error:

Failure happened on 'Source' side. ErrorCode=RestCallFailedWithClientError,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=Rest call failed with client error, status code 400 BadRequest, please check your activity settings.

Request URL: https://my.api.com/transactions?<suppressed 1 parameter(s)>.

Response: {"success":false,"message":"Invalid cursor"},Source=Microsoft.DataTransfer.ClientLibrary,'

I have implemented a workaroud using the Until Activit to parse through the various pages but that is not ideal as it takes about 30-45 seconds for each page which adds up quickly. This however confirms that the API call works as expected when extracting the cursor.next value and passing it as input to the API to retreive the next page.

Any guidance on this will be much appreciated. I suspect there is something minor that needs to be tweaked to fix this.

[fabric-dev79]

Thank you @nilendraFabric. Your suggestion solved my problem. Much appreciated

[nilendraFabric · Accepted solution]

Hello @fabric-dev79 

Try Using the base endpoint without the cursor parameter:
`https://my.api.com/transactions`
2. Pagination Rules:
• QueryParameters.cursor:
Set to `$.cursor.next` (JSON path to the cursor value in the API response).
• EndCondition:
Use `$.cursor.next == null` to stop pagination when no more pages exist.

https://learn.microsoft.com/en-us/azure/data-factory/connector-rest?tabs=data-factory