Forum Discussion

BI007's avatar
BI007
Helper I
1 year ago
Solved

Dynamic data source issue

Hello. I have this M script in Power Query, which is getting data using api (using pagination): let // Function to retrieve a page of data GetPage = (url as text) => let RawData = Json.Doc...
  • Anonymous's avatar
    Anonymous
    1 year ago

    Hi BI007 ,
    Thanks for ZhangKun reply.
    One way to do this is to make the URL parameters static in the query folding context

    let
        GetPage = (url as text) =>
        let
            RawData = Json.Document(Web.Contents(url, [Headers=[Authorization="Bearer " & GetToken()]])),
            Users = RawData[value],
            NextLink = try RawData[#"@odata.nextLink"] otherwise null,
            DataTable = if List.IsEmpty(Users) then null else Table.FromList(Users, Splitter.SplitByNothing(), null, null, ExtraValues.Error)
        in
            [Table = DataTable, NextLink = NextLink],
        InitialUrl = "https://graph.microsoft.com/v1.0/groups",
        Initial = GetPage(InitialUrl),
        AllPages = List.Generate(
            () => [Page = Initial, Link = Initial[NextLink]],
            each [Link] <> null,
            each [Page = GetPage([Link]), Link = [Page][NextLink]],
            each [Page][Table]
        ),
    
        CombinedData = Table.Combine(List.RemoveNulls(AllPages)),
        #"Expanded Column1" = Table.ExpandRecordColumn(CombinedData, "Column1", {"id", "description"}, {"id", "description"})
    in
        #"Expanded Column1"
    

    You can also use the RelativePath and Query options to build more flexible and maintainable URLs.

    let
        GetPage = (relativePath as text, query as record) =>
        let
            RawData = Json.Document(Web.Contents("https://graph.microsoft.com", [RelativePath = relativePath, Query = query, Headers = [Authorization = "Bearer " & GetToken()]])),
            Users = RawData[value],
            NextLink = try RawData[#"@odata.nextLink"] otherwise null,
            DataTable = if List.IsEmpty(Users) then null else Table.FromList(Users, Splitter.SplitByNothing(), null, null, ExtraValues.Error)
        in
            [Table = DataTable, NextLink = NextLink],
        InitialRelativePath = "v1.0/groups",
        InitialQuery = [],
        Initial = GetPage(InitialRelativePath, InitialQuery),
        AllPages = List.Generate(
            () => [Page = Initial, Link = Initial[NextLink]],
            each [Link] <> null,
            each [Page = GetPage(Text.Middle([Link], Text.Length("https://graph.microsoft.com/")), []), Link = [Page][NextLink]],
            each [Page][Table]
        ),
    
        CombinedData = Table.Combine(List.RemoveNulls(AllPages)),
        #"Expanded Column1" = Table.ExpandRecordColumn(CombinedData, "Column1", {"id", "description"}, {"id", "description"})
    in
        #"Expanded Column1"

    Data refresh in Power BI - Power BI | Microsoft Learn

     

    Best regards,
    Albert He


    If this post helps, then please consider Accept it as the solution to help the other members find it more quickly