Forum Discussion
List.ParallelInvoke strange behaviour
Hi,
We are trying to use List.ParallelInvoke, in custom connector project, which calls Rest Api for fetching data.
When we pass parameters there is a hang.
shared ParallelInvoke.Contents = Value.ReplaceType(ParallelInvokeImpl, ParallelInvokeType); or shared ParallelInvoke.Contents = (a as text) => NavTable() as table; we get hang of our connector.
In case of shared ParallelInvoke.Contents = () => NavTable() as table; it works fine.
ParallelInvokeType = type function (a as (type text meta [
Documentation.FieldCaption = "Frequency",
Documentation.AllowedValues = { "Monthly", "Daily", "Hourly" },
Documentation.FieldDescription = "Defines the sampling rate in which the data is retrieved from the source"
]))
as table meta [
// appears in the initial datasource parameter configuration dialog
Documentation.Name = "ABB Motion API parameters"
];
ParallelInvokeImpl = (a as text) =>
let
r = NavTable()
in
r;
Maybe you can help us, and tell, what can be the reason of this behaviour? artemus
4 Replies
I don't think that function is offically supported. That being said, I think it needs to be invoked on a list stored in a deferred evaulation context like a record or a table.
We use it like this:
parallelRequest = () as table =>
let
l = {1 .. 30},
lf = List.Transform(l, each () => GetParallelData(_)),
p2 = List.ParallelInvoke({ () => GetParallelData(2), () => GetParallelData(3)}),
p = List.ParallelInvoke(lf, 10),
r = Table.Combine(p)
in
r;GetParallelData = (postNumber as number) as table =>
let
Source = Json.Document(Web.Contents("https://jsonplaceholder.typicode.com/posts/" & Number.ToText(postNumber), [Headers=[Authorization="Bearer " & "asd"], ManualStatusHandling={400, 401, 403, 404, 429, 500, 503}])),
table = Record.ToTable(Source),
pivot = Table.Pivot(table, List.Distinct(table[Name]), "Name", "Value"),
r = pivot
in
r;NavTable = () as table =>
let
source = #table({"Name", "Data", "ItemKind", "ItemName", "IsLeaf"}, {
{ "parallelRequest", parallelRequest(), "Table", "Table", true }
}),
navTable = Table.ToNavigationTable(source, {"Name"}, "Name", "Data", "ItemKind", "ItemName", "IsLeaf")
in
navTable;1. Please write example, or correct our code...how to implement? artemus
2. Is there other methods in Power Query, for parallelizm? artemus
I think it *might* work (I haven't tried this myself) if you change a line from:
p2 = List.ParallelInvoke({ () => GetParallelData(2), () => GetParallelData(3)}),
to
p2 = List.ParallelInvoke([f = { () => GetParallelData(2), () => GetParallelData(3)}][f]),
2. Power query will generally execute items in parellel if they can be lazly loaded. Values in Records and Tables are loaded lazly. Also note that running your connector in Visual studio works a bit differently than in Power Bi, in order to make debugging easier. If you aren't doing so already, test your connector in Power Bi (Power Bi will reload your connector every time it is modified).
