Forum Discussion
script running looping URLs and get data
- 9 years ago
When you add a custom column (Invoke Custom Function), change the drop down to Column Name and select the app_id column. Then, the function will pass in the app_id value from each row into the function (into the URL) and perform the query steps you designed earlier. If you need to pass a bearer token to the URL for each app_id value, if it is the same bearer token for every app_id, you can add an Authorization header to the URL request.
For example, I have a dashboard that pulls data from Twitter. Once I obtain the access token (it is the result of a different query named AccessToken) here is the function I use to query Twitter:
(params) => let GetJsonQuery = Web.Contents("https://api.twitter.com/1.1/search/tweets.json" & params, [ Headers = [#"Authorization"=AccessToken] ] ), FormatAsJsonQuery = Json.Document(GetJsonQuery), data = try FormatAsJsonQuery[statuses] otherwise null, next = try FormatAsJsonQuery[search_metadata][next_results] otherwise null, res = [Data=data, Next=next] in resI have a table with a column of search words, where the column name is "keyword". The parameter named "params" that started off the function above is defined as:
params = "?q=" & keyword & "&count=100"
So you can see how the column of hard coded search terms is passed into the url part defined by "params" which is in turn passed into the function. The function runs on every row in my table of "keyword", so each row returns a table of results, which I can then expand and Power BI will automatically append the results into one table.
Note: This is not the entire query and will not return results (the "data", "next" and "res" variables are used in another query that iterates through pages of results). Hopefully this shows you how to structure your web call using the bearer token.
If you have a list of URLs ahead of time, you can load that into Power BI as a table with URL as a column. Then, as a separate query, connect to an individual URL using the web connector and add the appropriate headers to the M code. Continue through the query design process (i.e. renaming columns, data types, transformations etc.). Once completed, right click on this query and select "Convert to Function". This will take all the steps you performed and convert to a function that can be called for every URL in the table you loaded earlier. Once it is a function, you will need to edit the start of the code so it looks like the following:
myFunction = (column) as table =>
let
source = column,
After this source step, your next step should be the web contents step, and you will replace the URL that was hard coded there with "source" (no quotes). Go to the table of URLs and add a column > "Invoke Custom Function" and choose the function myFunction and from the other drop down select the column name of the URLs column.
If you need more detail on how to do what I described, let me know.