Rest API Pagination Issue

[NAkhliesh]

We are using Copy Activity for getting data from Oracle P6 Rest API. But we are getting only 5000 records due to pagination.

While using QueryParameter,we are getting duplicate data of the first 5000 records multiple times.

Could you please help me to get Incremental data.

Over alternative approach the bearer token is not getting generated using Web Activity.
Need help on this.

[NAkhliesh]

Hi Xiaoxin Sheng,

The Url and Relative Url we are using is these:

https://ksa1.p6.oraclecloud.com

/xxxxxx/pds/rest-service/dataservice/runquery?configCode=ds_p6adminuser

and provided the pagination as below:

and the output is we are getting 40 times duplicate data.

Could you help us to get the Incremental data.

[FabianSchut · Accepted solution]

I don't see any variable used in your relative url which holds the pagination. You should follow the documentation which url  parameters you should use. But, let's say you have two url parameters offset and count. Your relative url should be

service/dataservice/runquery?configCode=ds_p6adminuser&offset=offsetVariable&count=5000. 
You should have created a variable named offsetVariable and count should be equal to the last column of your pagination rules, which is also 5000 in your screenshot. You should change {offset} to the variable offsetVariable. 

[v-shex-msft]

HI @NAkhliesh,

Can you please share some more detailed information about this api and its definitions? They should help us clarify your scenario and test.

BTW, if you not clear how to use copy activity with pagination api, you can also try to invoke notebook and use pyspark code to handle these. 

Reference link:

pagination rule in copy activity

Configure REST in a copy activity - Microsoft Fabric | Microsoft Learn

Regards,

Xiaoxin Sheng

[FabianSchut]

Hi, I think your parameter offset is not set correctly. What is your relative URL and are you using parameters for this setup? I found this post helpful in understanding the necessary steps: https://techcommunity.microsoft.com/blog/fasttrackforazureblog/implementing-pagination-with-the-copy.... Please read step 2 carefully and check if that setup is used correctly.