topic Re: A generic REST API connector? in Developer

A generic REST API connector?

mrgou — Fri, 12 Mar 2021 10:33:19 GMT

I need to query data from a REST API but couldn't find any appropriate connector, or even a generic connector. Such a connector should be able to handle requests in any HTTP method, manage authentication by passing a user name and password as parameters of a POST request and using the returned token in the header of subsequent requests, handle pagination of results (i.e. looping queries until there's no next page)...

A generic connector may not be possible, and I may need to create a custom connector, but it's a steep learning curve, so I was hoping for at least something that I can start from and modify.

Any hint?

Re: A generic REST API connector?

Anonymous — Mon, 15 Mar 2021 07:09:38 GMT

Hi @mrgou,

You can try to use the following M query code if they work on your side:

let rooturl = "https://myserver.com/", //profile id oath_oidc_profile_id = "xxxxxx", //access token from your portal configurations token = "xxxxxxx", relativeURL = "auth/oauth/session/" & oath_oidc_profile_id, //connect to server GetJson = Web.Contents( rooturl, [ Headers = [ #"Authorization" = "Bearer" & token, #"Content-Type" = "application/x-www-form-urlencoded" ], RelativePath = relativeURL ] ), //'session Id' for advanced operation sessionId = Json.Document(GetJson)[sessionId], Source = Web.Contents( rooturl, [ Headers = [ #"Authorization" = "Bearer" & token, #"Content-Type" = "application/x-www-form-urlencoded" ], RelativePath = "xxxx/xxxxx/xxxxx", Content = Text.ToBinary("client_id=" & sessionId) ] ) in Source

If the above not help, you can also take a look at the following link about pull data from the rest API with authentication:

Pull data from a REST API Authentication
Regards,

Xiaoxin Sheng

Re: A generic REST API connector?

mrgou — Tue, 16 Mar 2021 11:22:07 GMT

@Anonymous Thank you for sharing this. I looked into the M reference, and also found a WebAction.Request method that appears to allow using other HTTP methods. I guess I'll have to go ahead and study the language. Unfortunately, I couldn't find any good learning resource, so I'll have to do it the hard way with the specification and reference documentation from Microsoft!

Re: A generic REST API connector?

Anonymous — Wed, 17 Mar 2021 06:15:14 GMT

HI @mrgou,

If you can't find enough code sample/tutorials of power query functions. You can also take a look at Chris Webb's blog, he shares lots of blogs of power query function usage:

Chris Webb's BI Blog # Web.Contents
Regards,

Xiaoxin Sheng

Re: A generic REST API connector?

Anonymous — Tue, 29 Mar 2022 21:48:16 GMT

Hi,

I have two questions,

First, It is my understanding this would only work locally and not in the service. As it will treat "Web.Contents(<url variable>" in this case "Web.Contents(rooturl" as a dynamic connector and thus will not refresh. Changing it to Web.Contents("https://myserver.com/" .. would result in the service removing this error. The service might still show the "invalid credentials error". According to Chris Webb this can be fixed in code, but i am not sure how. The easier option is just to turn off the credential check in the service. It does need more testing but it seems to work.

Last question, is around this line here:

sessionId = Json.Document(GetJson)[sessionId]

I had a similar code:

Json = GetJson(Url), //Getjson return a Json.Document. Value = Json[#"tickets"]

Which I tried to parameterize.

//this works Value = Json[tickets] //but this does not: var = "tickets" Value = Json[var] //ended up with: var = "tickets" Value = Record.FieldValues(Record.SelectFields(Json, var)){0}

There has got to be single or at least simplier method... right?

Re: A generic REST API connector?

sherodkeen — Wed, 02 Nov 2022 20:02:30 GMT

The code to generate the token seems to work great, except when I use it against an api endpoint where there are so many records that the token eventually expires. It seems that the code only uses the same token over and over.

See my issue here:

https://community.powerbi.com/t5/Power-Query/Bearer-Token-Expires-before-GET-Request-Completes/m-p/2847411/highlight/true#M88507

Re: A generic REST API connector?

Anonymous — Mon, 09 Oct 2023 00:40:59 GMT

This is not "generic" as it is specific for Zendesk. But it should be possible to make it generic https://community.fabric.microsoft.com/t5/Desktop/Custom-Zendesk-Connector-Power-Query-Code/td-p/2407620.