Forum Discussion
script running looping URLs and get data
- 9 years ago
When you add a custom column (Invoke Custom Function), change the drop down to Column Name and select the app_id column. Then, the function will pass in the app_id value from each row into the function (into the URL) and perform the query steps you designed earlier. If you need to pass a bearer token to the URL for each app_id value, if it is the same bearer token for every app_id, you can add an Authorization header to the URL request.
For example, I have a dashboard that pulls data from Twitter. Once I obtain the access token (it is the result of a different query named AccessToken) here is the function I use to query Twitter:
(params) => let GetJsonQuery = Web.Contents("https://api.twitter.com/1.1/search/tweets.json" & params, [ Headers = [#"Authorization"=AccessToken] ] ), FormatAsJsonQuery = Json.Document(GetJsonQuery), data = try FormatAsJsonQuery[statuses] otherwise null, next = try FormatAsJsonQuery[search_metadata][next_results] otherwise null, res = [Data=data, Next=next] in resI have a table with a column of search words, where the column name is "keyword". The parameter named "params" that started off the function above is defined as:
params = "?q=" & keyword & "&count=100"
So you can see how the column of hard coded search terms is passed into the url part defined by "params" which is in turn passed into the function. The function runs on every row in my table of "keyword", so each row returns a table of results, which I can then expand and Power BI will automatically append the results into one table.
Note: This is not the entire query and will not return results (the "data", "next" and "res" variables are used in another query that iterates through pages of results). Hopefully this shows you how to structure your web call using the bearer token.
If you have a list of URLs ahead of time, you can load that into Power BI as a table with URL as a column. Then, as a separate query, connect to an individual URL using the web connector and add the appropriate headers to the M code. Continue through the query design process (i.e. renaming columns, data types, transformations etc.). Once completed, right click on this query and select "Convert to Function". This will take all the steps you performed and convert to a function that can be called for every URL in the table you loaded earlier. Once it is a function, you will need to edit the start of the code so it looks like the following:
myFunction = (column) as table =>
let
source = column,
After this source step, your next step should be the web contents step, and you will replace the URL that was hard coded there with "source" (no quotes). Go to the table of URLs and add a column > "Invoke Custom Function" and choose the function myFunction and from the other drop down select the column name of the URLs column.
If you need more detail on how to do what I described, let me know.
Hey dkay84_PowerBI ,
thanks a lot for the solution, but iam facing an issue after all this. The function i created returns all the tables from the list of URLs but when i expand and click on close and apply , It throws an error called "Access to the resource is forbidden" , I tried clearing out all permissions multiple times but resulted in same thing
Ps:I am passing my bearer token along with the url in headers
Regards
Yeswanth