Problem with GitLab API pagination in Power BI - Limited response size

[Anonymous]

Hello everyone,

I am facing a problem with the integration of the GitLab API in Power BI. I am using the GitLab API to retrieve issues from our company and I am running into a limitation with pagination. Although there are about 5400 issues in total in our GitLab:

I only ever get 258 issues back via the API queries in Power BI (after adjusting the per_page header. Otherwise there would only be 20 records in response due to the default GitLab API return limit).

Even when docking the endpoint via Postman with keyset pagination, I only get 258 for X-Total:

running this GET request: https://gitlab.company_name.com/api/v4/issues?pagination=keyset&per_page=2&order_by=updated_at&sort=...

The Power-Query Script im currently using in the Dataflow:

But Response column is just 258 records as well: 

I appreciate any help!

Best regards 🙂

[Anonymous]

Hi @Anonymous 

Based on your description, you can refer to the following link.

curl - Pagination in Gitlab API only returns 100 per_page max - Stack Overflow

Solved: API limited to 50 responses - pagination looping r... - Microsoft Fabric Community

Best Regards!

Yolo Zhu

If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.

[Anonymous]

Thanks for your quick answer!

In my described approach to the power query script i already tried to call the API iteratively until i get back pages without content. but so far this only works so well that i get 258 records as return instead of 100 (because 100 is the max number of returns per page). And don't know at all why that is because the total issue number on the gitlab server is about 5k.

[Anonymous · Accepted solution]

Fixed it. Needed to set the header parameter "scope=all" as described in the api docs for gitlab: https://docs.gitlab.com/ee/api/issues.html 

[SaxonChuang]

I my case the parameter scope=all is not work. here is a example for get all issues via power query.

let
    // Base API URL (Replace {project_id} with your actual project ID)
    BaseUrl = "https://gitlab.com/api/v4/projects/535/issues?per_page=100",
    
    // Headers including the scope
    Headers = [
        #"PRIVATE-TOKEN" = "........."
    ],

    // get single page
    GetPage = (PageNum as number) =>
        let
            Url = BaseUrl & "&page=" & Number.ToText(PageNum),
            Response = Web.Contents(Url, [Headers = Headers]),
            JsonData = Json.Document(Response)
        in
            JsonData,

    // loop all pages
    GetAllIssues = List.Generate(
        () => [PageNum = 1, Issues = GetPage(1)],  // 開始從第 1 頁
        each List.Count(_[Issues]) > 0,           // 只要回傳不為空，繼續抓取
        each [PageNum = _[PageNum] + 1, Issues = GetPage(_[PageNum] + 1)],  // 取下一頁
        each _[Issues]                            // 取出 issue 清單
    ),

    // flat them
    FlattenedIssues = List.Combine(GetAllIssues),

    // turn to table
    IssuesTable = Table.FromList(FlattenedIssues, Splitter.SplitByNothing(), null, null, ExtraValues.Error)

in
  IssuesTable