Forum Discussion
Help with recursive web power query (oDATA, SuccessFactors)
- 5 years ago
Here is an example on how to do this with List.Generate using a test REST API. I adapted the approach described well in this article. You can paste the code into a blank query to step through it.
List.Generate() and Looping in PowerQuery - Exceed
let fn = (pagenumber) => Json.Document(Web.Contents("https://api.instantwebtools.net/v1/passenger?page=" & pagenumber & "&size=1000")), mylist = List.Generate(()=> [Page = 1, Result = fn(Number.ToText(1))], each List.Count([Result][data])>0, each [Page = [Page] + 1, Result = fn(Number.ToText([Page]+1))]), #"Converted to Table" = Table.FromList(mylist, Splitter.SplitByNothing(), null, null, ExtraValues.Error), #"Expanded Column1" = Table.ExpandRecordColumn(#"Converted to Table", "Column1", {"Page", "Result"}, {"Page", "Result"}), #"Expanded Result" = Table.ExpandRecordColumn(#"Expanded Column1", "Result", {"data"}, {"data"}), #"Expanded data" = Table.ExpandListColumn(#"Expanded Result", "data"), #"Expanded data1" = Table.ExpandRecordColumn(#"Expanded data", "data", {"_id", "name", "trips", "airline", "__v"}, {"_id", "name", "trips", "airline", "__v"}) in #"Expanded data1"Pat
- 5 years ago
Hello Pat, thank you very much for your help!
You inspired me and I was able to contruct my own solution. It seems very easy at the first sight and also works great for my scenario:
This returns a list of lists of records that I can easily expand and work with.
It works great, loads data - but maybe looks too simple when compared with other examples of recursive functions I have been going through.
Could you please have a look at my code if you have any possible concerns about it, because I can't believe this short code actually does tone of work.
Thanks, Jakub!
Hello Pat, thank you very much for your help!
You inspired me and I was able to contruct my own solution. It seems very easy at the first sight and also works great for my scenario:
This returns a list of lists of records that I can easily expand and work with.
It works great, loads data - but maybe looks too simple when compared with other examples of recursive functions I have been going through.
Could you please have a look at my code if you have any possible concerns about it, because I can't believe this short code actually does tone of work.
Thanks, Jakub!
Picking up on this - as for our organization - and for security in general - we did not want to use basic authentication against SuccessFactors - I came up with an alternative to the marked solution here.
Firstly I created a seperate dataflow for the token authentication.
For that part I use a service account and generate the assertion with the SAML generator supplied from SAP.
For the powerquery for this part it looks like below:
2nd - for any new Odata sources I want to fetch into PowerBI I setup a seperate dataflow, where I reference the Authentication Token ( this is running 1 a day to get a new token)
For getting the odata - I have this function. For this to work, you should always use inlinecount=allpages in the query.