Forum Discussion
Get data from web error: A web API key can only be specified when a web API key name is provided
Ok, since you're going down the rabbit hole :) Here's some M code you should be able to adapt to do what you need. It's taken from a different project I worked on some time ago so it will show you the way, but you'll need to craft your solution. It relies on using M queries as functions.
TPalmer might be able to help here too.
Table.GenerateByPage = (getNextPage as function) as table =>
let
listOfPages = List.Generate(
() => getNextPage(null),
(lastPage) => lastPage <> null,
(lastPage) => getNextPage(lastPage)
),
tableOfPages = Table.FromList(listOfPages, Splitter.SplitByNothing(), {"Column1"}),
firstRow = tableOfPages{0}?
in
if (firstRow = null) then
Table.FromRows({})
else
Value.ReplaceType(
Table.ExpandTableColumn(tableOfPages, "Column1", Table.ColumnNames(firstRow[Column1])),
Value.Type(firstRow[Column1])
),
GetNextLink = (link) =>
let
links = Text.Split(link, ","),
splitLinks = List.Transform(links, each Text.Split(Text.Trim(_), ";")),
next = List.Select(splitLinks, each Text.Trim(_{1}) = "rel=""next"""),
first = List.First(next),
removedBrackets = Text.Range(first{0}, 1, Text.Length(first{0}) - 2)
in
try removedBrackets otherwise null,
GetContents = (url as text) =>
let
content = Web.Contents(url),
link = GetNextLink(Value.Metadata(content)[Headers][#"Link"]?),
json = Json.Document(content)
in
json meta [Next=link],
GetPagedTable = (url as text) => Table.GenerateByPage((previous) =>
let
next = if previous = null then null else Value.Metadata(previous)[Next],
current = if previous <> null and next = null then null else GetContents(if next = null then url else next),
link = if current = null then null else Value.Metadata(current)[Next],
table = if current = null then null else Table.FromList(current, Splitter.SplitByNothing(), null, null, ExtraValues.Error)
in
table meta [Next=link])
At the end of the day you'll call
Source = GetPagedTable( <<your url>>),
HTH, -Lukasz
Hi Lukasz
I am trying to do something similar. I am trying to query the MS Cognitive Services Text Analytics API via the function below. As that API is asynchronous I need to extract the Operation-Location header value in order to retrieve the results. However, the Web.Contents function seems to only return Response Status.
For testing, I have modified the function to return the headers for the response metadata as per your example code but all I get is Content-Type field with a null value. Am I right in summising that it is not possible to extract response header data (except for status) and therefore this sort of processing is only possible if link/location values are retuned as content rather than in headers?
(Source as table) as any =>
let
JsonRecords = Text.FromBinary(Json.FromValue(Source)),
JsonRequest = "{ ""stopWords"": [], ""topicsToExclude"": [],""documents"": " & JsonRecords & "}",
JsonContent = Text.ToBinary(JsonRequest, TextEncoding.Ascii),
Response =
Web.Contents("https://westus.api.cognitive.microsoft.com/text/analytics/v2.0/topics",
[
Headers = [#"Ocp-Apim-Subscription-Key"= "xxxxxxxxxxxxxxxxxxxxxxxxxx",
#"Content-Type"="application/json"],
Content=JsonContent,
ManualStatusHandling={202}
]),
ResponseMeta = Value.Metadata(Response),
ResponseStatus = ResponseMeta[Response.Status],
Output = if ResponseStatus = 200 then Json.Document(Response,1252) else ResponseMeta[Headers]
in
Output