Forum Discussion
script running looping URLs and get data
- 9 years ago
When you add a custom column (Invoke Custom Function), change the drop down to Column Name and select the app_id column. Then, the function will pass in the app_id value from each row into the function (into the URL) and perform the query steps you designed earlier. If you need to pass a bearer token to the URL for each app_id value, if it is the same bearer token for every app_id, you can add an Authorization header to the URL request.
For example, I have a dashboard that pulls data from Twitter. Once I obtain the access token (it is the result of a different query named AccessToken) here is the function I use to query Twitter:
(params) => let GetJsonQuery = Web.Contents("https://api.twitter.com/1.1/search/tweets.json" & params, [ Headers = [#"Authorization"=AccessToken] ] ), FormatAsJsonQuery = Json.Document(GetJsonQuery), data = try FormatAsJsonQuery[statuses] otherwise null, next = try FormatAsJsonQuery[search_metadata][next_results] otherwise null, res = [Data=data, Next=next] in resI have a table with a column of search words, where the column name is "keyword". The parameter named "params" that started off the function above is defined as:
params = "?q=" & keyword & "&count=100"
So you can see how the column of hard coded search terms is passed into the url part defined by "params" which is in turn passed into the function. The function runs on every row in my table of "keyword", so each row returns a table of results, which I can then expand and Power BI will automatically append the results into one table.
Note: This is not the entire query and will not return results (the "data", "next" and "res" variables are used in another query that iterates through pages of results). Hopefully this shows you how to structure your web call using the bearer token.
The only part that changes is the middle part to identify each app, with unique random number (e.g., https://...3/...). Thanks.
Can you give a full example of what the URL looks like?
Essentially, what I am trying to understand is the strucuture of the URL so that we can figure out the best approach. Ultimately, what we will end up doing is turning your query into a function and invoking that function on a table of your unique URLs, but how we get to that table is the wild card.
- Anonymous9 years agoNot applicable
Here they are:
https://api.appannie.com/v1.2/intelligence/apps/ios/app/310738695/history?countries=US&feeds=downloads&granularity=monthly&device=all&start_date=2013-09-01&end_date=2017-02-28
https://api.appannie.com/v1.2/intelligence/apps/ios/app/215034499/history?countries=US&feeds=downloads&granularity=monthly&device=all&start_date=2013-09-01&end_date=2017-02-28
with Header, Authorization bearer <key> Hope this clarifies things. Thanks.
- dkay84_PowerBI9 years agoMicrosoft Employee
Yes thank you. And you said that the number in the middle (bold text) is randomly generated, meaning we can't have a list of these in advance? If so, how do you get the URL in the first place? Is there an earlier query that gets this information?
- Anonymous9 years agoNot applicable
THe list of these URLs will be generated in advace (most likely XLS), then this looping script will do process the URLs and get responses back. Thanks.