Forum Discussion

jdusek92's avatar
jdusek92
Advocate III
5 years ago
Solved

Help with recursive web power query (oDATA, SuccessFactors)

Hello, I am connecting to oDATA SuccessFactors and I have encountered the following challenge:   I can get only 1000 rows in one JSON file (default limitation). With these 1000 records, I also get...
  • mahoneypat's avatar
    5 years ago

    Here is an example on how to do this with List.Generate using a test REST API.  I adapted the approach described well in this article.  You can paste the code into a blank query to step through it. 

     

    List.Generate() and Looping in PowerQuery - Exceed

     

    let
        fn = (pagenumber) => Json.Document(Web.Contents("https://api.instantwebtools.net/v1/passenger?page=" & pagenumber & "&size=1000")),
        mylist = List.Generate(()=> [Page = 1, Result = fn(Number.ToText(1))], each List.Count([Result][data])>0, each [Page = [Page] + 1, Result = fn(Number.ToText([Page]+1))]),
        #"Converted to Table" = Table.FromList(mylist, Splitter.SplitByNothing(), null, null, ExtraValues.Error),
        #"Expanded Column1" = Table.ExpandRecordColumn(#"Converted to Table", "Column1", {"Page", "Result"}, {"Page", "Result"}),
        #"Expanded Result" = Table.ExpandRecordColumn(#"Expanded Column1", "Result", {"data"}, {"data"}),
        #"Expanded data" = Table.ExpandListColumn(#"Expanded Result", "data"),
        #"Expanded data1" = Table.ExpandRecordColumn(#"Expanded data", "data", {"_id", "name", "trips", "airline", "__v"}, {"_id", "name", "trips", "airline", "__v"})
    in
        #"Expanded data1"

     

    Pat

  • jdusek92's avatar
    jdusek92
    5 years ago

    Hello Pat, thank you very much for your help!

     

    You inspired me and I was able to contruct my own solution. It seems very easy at the first sight and also works great for my scenario:

     

    This returns a list of lists of records that I can easily expand and work with.

     

    It works great, loads data - but maybe looks too simple when compared with other examples of recursive functions I have been going through.

    Could you please have a look at my code if you have any possible concerns about it, because I can't believe this short code actually does tone of work.

     

    Thanks, Jakub!