Forum Discussion
Help with recursive web power query (oDATA, SuccessFactors)
- 5 years ago
Here is an example on how to do this with List.Generate using a test REST API. I adapted the approach described well in this article. You can paste the code into a blank query to step through it.
List.Generate() and Looping in PowerQuery - Exceed
let fn = (pagenumber) => Json.Document(Web.Contents("https://api.instantwebtools.net/v1/passenger?page=" & pagenumber & "&size=1000")), mylist = List.Generate(()=> [Page = 1, Result = fn(Number.ToText(1))], each List.Count([Result][data])>0, each [Page = [Page] + 1, Result = fn(Number.ToText([Page]+1))]), #"Converted to Table" = Table.FromList(mylist, Splitter.SplitByNothing(), null, null, ExtraValues.Error), #"Expanded Column1" = Table.ExpandRecordColumn(#"Converted to Table", "Column1", {"Page", "Result"}, {"Page", "Result"}), #"Expanded Result" = Table.ExpandRecordColumn(#"Expanded Column1", "Result", {"data"}, {"data"}), #"Expanded data" = Table.ExpandListColumn(#"Expanded Result", "data"), #"Expanded data1" = Table.ExpandRecordColumn(#"Expanded data", "data", {"_id", "name", "trips", "airline", "__v"}, {"_id", "name", "trips", "airline", "__v"}) in #"Expanded data1"Pat
- 5 years ago
Hello Pat, thank you very much for your help!
You inspired me and I was able to contruct my own solution. It seems very easy at the first sight and also works great for my scenario:
This returns a list of lists of records that I can easily expand and work with.
It works great, loads data - but maybe looks too simple when compared with other examples of recursive functions I have been going through.
Could you please have a look at my code if you have any possible concerns about it, because I can't believe this short code actually does tone of work.
Thanks, Jakub!
Hello Pat, thank you very much for your help!
You inspired me and I was able to contruct my own solution. It seems very easy at the first sight and also works great for my scenario:
This returns a list of lists of records that I can easily expand and work with.
It works great, loads data - but maybe looks too simple when compared with other examples of recursive functions I have been going through.
Could you please have a look at my code if you have any possible concerns about it, because I can't believe this short code actually does tone of work.
Thanks, Jakub!
If it works and is performant, that's the real test. Does it make two web calls with each iteration?
Pat
- jdusek925 years agoAdvocate III
How can I test that it does not perform all the calls twice?
Warm regards, Jakub
- mahoneypat5 years agoMicrosoft Employee
You could probably tell with the query diagnostics features (or with an external tool like Fiddler); however, if it is performant and you don't have a limit on web calls, don't worry about it.
Pat