Forum Discussion

Anonymous's avatar
Anonymous
Not applicable
2 years ago
Solved

Problem with GitLab API pagination in Power BI - Limited response size

Hello everyone,   I am facing a problem with the integration of the GitLab API in Power BI. I am using the GitLab API to retrieve issues from our company and I am running into a limitation with pag...
Need Help
Anonymous's avatar
Anonymous
Not applicable
2 years ago

Thanks for your quick answer!

 

In my described approach to the power query script i already tried to call the API iteratively until i get back pages without content. but so far this only works so well that i get 258 records as return instead of 100 (because 100 is the max number of returns per page). And don't know at all why that is because the total issue number on the gitlab server is about 5k.

Anonymous's avatar
Anonymous
Not applicable
2 years ago

Fixed it. Needed to set the header parameter "scope=all" as described in the api docs for gitlab: https://docs.gitlab.com/ee/api/issues.html 

  • SaxonChuang's avatar
    SaxonChuang
    New Member
    1 year ago

    I my case the parameter scope=all is not work. here is a example for get all issues via power query.

     

    let
    // Base API URL (Replace {project_id} with your actual project ID)
    BaseUrl = "https://gitlab.com/api/v4/projects/535/issues?per_page=100",
    
    // Headers including the scope
    Headers = [
        #"PRIVATE-TOKEN" = "........."
    ],

    // get single page
    GetPage = (PageNum as number) =>
        let
            Url = BaseUrl & "&page=" & Number.ToText(PageNum),
            Response = Web.Contents(Url, [Headers = Headers]),
            JsonData = Json.Document(Response)
        in
            JsonData,

    // loop all pages
    GetAllIssues = List.Generate(
        () => [PageNum = 1, Issues = GetPage(1)],  // 開始從第 1 頁
        each List.Count(_[Issues]) > 0,           // 只要回傳不為空，繼續抓取
        each [PageNum = _[PageNum] + 1, Issues = GetPage(_[PageNum] + 1)],  // 取下一頁
        each _[Issues]                            // 取出 issue 清單
    ),

    // flat them
    FlattenedIssues = List.Combine(GetAllIssues),

    // turn to table
    IssuesTable = Table.FromList(FlattenedIssues, Splitter.SplitByNothing(), null, null, ExtraValues.Error)

in
  IssuesTable