Forum Discussion

Anonymous's avatar
Anonymous
Not applicable
1 year ago
Solved

Rest API Pagination Issue

We are using Copy Activity for getting data from Oracle P6 Rest API. But we are getting only 5000 records due to pagination. While using QueryParameter,we are getting duplicate data of the first 500...
one lake
  • FabianSchut's avatar
    FabianSchut
    1 year ago

    I don't see any variable used in your relative url which holds the pagination. You should follow the documentation which url  parameters you should use. But, let's say you have two url parameters offset and count. Your relative url should be

    service/dataservice/runquery?configCode=ds_p6adminuser&offset=offsetVariable&count=5000. 
    You should have created a variable named offsetVariable and count should be equal to the last column of your pagination rules, which is also 5000 in your screenshot. You should change {offset} to the variable offsetVariable. 

Anonymous's avatar
Anonymous
Not applicable
1 year ago

HI Anonymous,

Can you please share some more detailed information about this api and its definitions? They should help us clarify your scenario and test.

BTW, if you not clear how to use copy activity with pagination api, you can also try to invoke notebook and use pyspark code to handle these. 

Reference link:

pagination rule in copy activity

Configure REST in a copy activity - Microsoft Fabric | Microsoft Learn

Regards,

Xiaoxin Sheng