Forum Discussion

nbj-ksdh's avatar
nbj-ksdh
Frequent Visitor
1 year ago
Solved

Fabric GraphQL Pagination

We will be using the new GraphQL API element to expose data from a Lakehouse to an application. The application needs to retrieve a lot of rows (+1mio). I can't find a single example online that showcases how to implement proper pagination.
Can anyone here assist with a query/script that loops through pages in batches of e.g. 1000 records?

  • Anonymous's avatar
    Anonymous
    1 year ago

    Hi nbj-ksdh ,
    Thank you for reaching out to Microsoft Fabric Community Forum!

    We sincerely apologize for the delay in response.Seems your issue is not resolved. Therefore, we request you to consider raising a support ticket to resolve the issue.

    To raise a support ticket for Fabric and Power BI, kindly follow the steps outlined in the following guide:

    How to create a Fabric and Power BI Support ticket - Power BI | Microsoft Learn

     

    If this post helps, please give us Kudos and consider marking it Accept as solution to assist other members in finding it more easily.

     

10 Replies

  • Hello nbj-ksdh 

     

    Please give this a try

     

    const fetch = require('node-fetch');

    // Example function to retrieve large sets in 1,000-row chunks.
    async function fetchAllRows() {
    let offset = 0;
    const limit = 1000;
    let allRows = [];

    while (true) {
    // Adjust field names or parameters as needed for your schema.
    const query = `
    query GetRows($limit: Int!, $offset: Int!) {
    myTable(limit: $limit, offset: $offset) {
    id
    name
    // Include other columns as necessary
    }
    }
    `;

    const response = await fetch("YOUR_GRAPHQL_ENDPOINT", {
    method: "POST",
    headers: { "Content-Type": "application/json" },
    body: JSON.stringify({ query, variables: { limit, offset } })
    });

    const json = await response.json();

    const currentBatch = json.data?.myTable || [];
    allRows.push(...currentBatch);

    // If there's fewer than 'limit' rows, we're done
    if (currentBatch.length < limit) {
    break;
    }

    offset += limit;
    }

    return allRows;
    }

    // Usage example:
    fetchAllRows()
    .then(rows => {
    console.log(`Retrieved ${rows.length} total rows.`);
    // Do something with your data here
    })
    .catch(err => console.error(err));

     

    function uses offset-based pagination (`limit` and `offset`), which is compatible with GraphQL implementations in Microsoft Fabric. However, if the schema includes fields like `endCursor` and `hasNextPage`, you might consider switching to cursor-based pagination for better efficiency

    You must create a GraphQL API endpoint within your Fabric workspace. This involves selecting a data source (e.g., a Data Warehouse or SQL database) and exposing the required tables or fields through the API. The schema is automatically generated based on the selected data source

    Please accept the answer if this is helpful and give kudos

    • nbj-ksdh's avatar
      nbj-ksdh
      Frequent Visitor

      nilendraFabric looks like you just pasted my request into ChatGPT and copied the answer here. If you have worked just a little with GraphQL in Fabric, you'd know that 'limit' and 'offset' aren't part of the schema...

      • nilendraFabric's avatar
        nilendraFabric
        Super User

        Apologies nbj-ksdh 

         

        I haven't used chatgpt as such , but yes haven't researched properly.


        You are correct here ,to implement pagination in Microsoft Fabric’s GraphQL API, you must use cursor-based pagination, as the schema does not support `limit` and `offset` parameters.

         

  • Anonymous's avatar
    Anonymous
    Not applicable

    Hi nbj-ksdh ,
    Thank you for reaching out to Microsoft Fabric Community Forum!

    We sincerely apologize for the delay in response.Seems your issue is not resolved. Therefore, we request you to consider raising a support ticket to resolve the issue.

    To raise a support ticket for Fabric and Power BI, kindly follow the steps outlined in the following guide:

    How to create a Fabric and Power BI Support ticket - Power BI | Microsoft Learn

     

    If this post helps, please give us Kudos and consider marking it Accept as solution to assist other members in finding it more easily.

     

  • Anonymous's avatar
    Anonymous
    Not applicable

    Hi nbj-ksdh ,
    Could you please confirm if the issue has been resolved after raising a support ticket? If a solution has been found, it would be greatly appreciated if you could share the insights with the community. This would be helpful for other members who may encounter similar issues.

    Thank you for being a valued member of the Microsoft Fabric Community Forum!

    Regards,
    Pallavi.

    • nbj-ksdh's avatar
      nbj-ksdh
      Frequent Visitor

      Hi Anonymous ,
      I have raised a ticket, but i haven't gotten an answer yet unfortunately. The issue persists. 

  • anders75's avatar
    anders75
    Regular Visitor

    Hello,

    We have an Object that we can query like: {  "query": "query { xxx_factgeneraljournalaccountentryv1s(first: 1000) { items {ledgeraccount accountingdate transactioncurrencyamount} } }"

    }
    Where can we change the GraphQL schema to support cursor-based pagination since the schema viewer in Fabric is read only?  

     

    ChatGPT tells me to ...

    1. Locate Your GraphQL Server Code:
    2. Find the file where your GraphQL schema is defined. This is typically a .graphql file or within your server code (e.g., using Apollo Server, Express-GraphQL, etc.).
    3. Add the Pagination Types:
      • Add the PageInfo type and other necessary types for cursor-based pagination to your schema definition.

         

     

    I dont know if this is the right way though.

    Any assistance is much appreciated.

    • Je-maintiendrai's avatar
      Je-maintiendrai
      Frequent Visitor

      What do you want to achieve? do you want to output more records? if so, this might work: change the number after first to your desired amount (up until 100.000). 

      query
      {
        publicholidays (first:10000, after:null){
           items {
              countryOrRegion
              holidayName
              countryRegionCode
              date
           }
        }
      }
       
      If you want more records add the endCursor function like below and use its output to requery the api (I dont have the entire way mapped out unfortunately since its not necessary for our company)

      query
      {
        publicholidays (first:10000, after:null){
           items {
              countryOrRegion
              holidayName
              countryRegionCode
              date
           }
           endCursor
           hasNextPage
        }
      }
  • Adjust the query below by changing the number after (first:

    query
    {
      publicholidays (first:10000, after:null){
         items {
            countryOrRegion
            holidayName
            countryRegionCode
            date
         }
      }
    }
     
    this worked for me, change the items queried to your data structure. Let me know if it also worked for you
  • Anonymous's avatar
    Anonymous
    Not applicable

    Hi nbj-ksdh ,

    We are following up once again regarding your query. Could you please confirm if the issue has been resolved through the support ticket with Microsoft?

    If the issue has been resolved, we kindly request you to share the resolution or key insights here to help others in the community. If we don’t hear back, we’ll go ahead and close this thread.

    Should you need further assistance in the future, we encourage you to reach out via the Microsoft Fabric Community Forum and create a new thread. We’ll be happy to help.

     

    Thank you for your understanding and participation.


    Regards,
    Pallavi G