topic Re: GraphQL- Shopify Pagination in Pipelines

GraphQL- Shopify Pagination

Siboska — Fri, 13 Jun 2025 11:00:19 GMT

Hi everyone,

I’m building an ADF pipeline to pull all orders from Shopify’s GraphQL API into an Azure SQL database, and I’m stuck on the pagination logic.

What I’ve tried so far

1. Use a Lookup activity to run the first GraphQL query (no after cursor) and extract

Store endCursor in a pipeline variable afterCursor.

2. Until loop

Condition: @equals( variables('hasNextPage'), false )

Copy (or Lookup) activity using a dynamic body that injects after:$afterCursor when it’s non-empty
Set variable → capture the new endCursor and hasNextPage from the JSON
Write out the nodes array to SQL

First-page query (Lookup):

@concat( '{', '"query":"query GetOrders($first:Int!,$filter:String){', 'orders(first:$first, sortKey:CREATED_AT, reverse:true, query:$filter){', 'nodes{',my fields....', '} ', 'pageInfo{hasNextPage endCursor}', '}', '}",', '"variables":{', '"first":250,', concat( '"filter":"created_at:>', variables('lastOrderDate'), '"' ), '}', '}' )

Loop query (Copy or Lookup inside Until):

@concat( '{', '"query":"query GetOrders($first:Int!,$after:String,$filter:String){', 'orders(first:$first', if( empty(variables('afterCursor')), '', concat(', after:\"', variables('afterCursor'), '\"') ), ', sortKey:CREATED_AT, reverse:true, query:$filter){', 'nodes{…} pageInfo{hasNextPage endCursor}', '} }",', '"variables":{', '"first":250', if( empty(variables('afterCursor')), '', concat(', "after":"', variables('afterCursor'), '"') ), concat( ', "filter":"created_at:>', coalesce(variables('lastOrderDate'),'1990-01-01T00:00:00Z'), '"' ), '}', '}' )

Where I’m stuck

I can extract the first endCursor outside the loop, but once inside the Until, I don’t know how to capture and reuse each subsequent endCursor that the Copy activity returns.
Copy activitys output only exposes metrics (rowsCopied, throughput, etc.), not the JSON body.

Can anyone advise:

A simpler pattern for Shopify GraphQL pagination in ADF?
How to “read” the next endCursor inside the loop (from a Copy or Lookup) so I can feed it back into the next iteration?

Thanks in advance for any guidance!

Re: GraphQL- Shopify Pagination

v-prasare — Mon, 16 Jun 2025 08:45:21 GMT

Hi Siboska

as per above query since you have mentioned the Copy activity does not contain the response body, so you are not able to access values like needed endCursor and hasNextPage for pagination. Instead, can you please try using a Lookup activity, which does include the full JSON response and allows you to extract the necessary data for paginated requests. please let us know if this helps you resolve your issue

Thanks,

Prashanth Are

MS Fabric Community Support

if you find this answer helpful please mark it accept as solution by doing so, this will help community members find asnwers quickly for similar issues.

Re: GraphQL- Shopify Pagination

Siboska — Mon, 16 Jun 2025 13:09:10 GMT

Thanks for the reply. I ended up going with the same solution.

For anyone else reading this: I had to wrap the initial load in an outer Lookup and Copy activity with a date-sorted filter to get the first batch of data. That way, I could use the resulting end cursor inside the Until loop. From my experience, setting the end cursor variable to null or blank isnt allowed.

Not sure if that’s the "correct" approach, but it worked for me.