Forum Discussion
Get data from web error: A web API key can only be specified when a web API key name is provided
Ok, since you're going down the rabbit hole :) Here's some M code you should be able to adapt to do what you need. It's taken from a different project I worked on some time ago so it will show you the way, but you'll need to craft your solution. It relies on using M queries as functions.
TPalmer might be able to help here too.
Table.GenerateByPage = (getNextPage as function) as table =>
let
listOfPages = List.Generate(
() => getNextPage(null),
(lastPage) => lastPage <> null,
(lastPage) => getNextPage(lastPage)
),
tableOfPages = Table.FromList(listOfPages, Splitter.SplitByNothing(), {"Column1"}),
firstRow = tableOfPages{0}?
in
if (firstRow = null) then
Table.FromRows({})
else
Value.ReplaceType(
Table.ExpandTableColumn(tableOfPages, "Column1", Table.ColumnNames(firstRow[Column1])),
Value.Type(firstRow[Column1])
),
GetNextLink = (link) =>
let
links = Text.Split(link, ","),
splitLinks = List.Transform(links, each Text.Split(Text.Trim(_), ";")),
next = List.Select(splitLinks, each Text.Trim(_{1}) = "rel=""next"""),
first = List.First(next),
removedBrackets = Text.Range(first{0}, 1, Text.Length(first{0}) - 2)
in
try removedBrackets otherwise null,
GetContents = (url as text) =>
let
content = Web.Contents(url),
link = GetNextLink(Value.Metadata(content)[Headers][#"Link"]?),
json = Json.Document(content)
in
json meta [Next=link],
GetPagedTable = (url as text) => Table.GenerateByPage((previous) =>
let
next = if previous = null then null else Value.Metadata(previous)[Next],
current = if previous <> null and next = null then null else GetContents(if next = null then url else next),
link = if current = null then null else Value.Metadata(current)[Next],
table = if current = null then null else Table.FromList(current, Splitter.SplitByNothing(), null, null, ExtraValues.Error)
in
table meta [Next=link])
At the end of the day you'll call
Source = GetPagedTable( <<your url>>),
HTH, -Lukasz
Thanks Lukas.
I must be doing something wrong with my zero level knowledge in M. Could not return any data.
Rather than learn M, I am trying to use a relatime API that is being developed. I get data returned but it takes ages to process each step after "source" - so far over 1 hr working on a "starts with" filter and the whole data is under 3500 rows.
- lukaszp9 years agoPower BI TeamNot sure what your API is doing. If it doesn't close the HTTP response, I bet the query tab is just sitting there waiting for the rest of the data.
- abedkhooli9 years agoFrequent Visitor
This seems very possible. I tried sending Connection: Close in the header but did not make a difference. My guess is that the API is ignoring it. I also used command timeout = 1.5 min (device reads every 60 sec). Waiting for feedback from the API developer.