Forum Discussion

Anonymous's avatar
Anonymous
Not applicable
6 years ago

List.ParallelInvoke strange behaviour

Hi, We are trying to use List.ParallelInvoke, in custom connector project, which calls Rest Api for fetching data. When we pass parameters there is a hang. shared ParallelInvoke.Contents = Value.R...
Need Help
artemus's avatar
artemus
Microsoft Employee
6 years ago

I don't think that function is offically supported. That being said, I think it needs to be invoked on a list stored in a deferred evaulation context like a record or a table.

  • Anonymous's avatar
    Anonymous
    Not applicable
    6 years ago

    We use it like this:


    parallelRequest = () as table =>
    let
    l = {1 .. 30},
    lf = List.Transform(l, each () => GetParallelData(_)),
    p2 = List.ParallelInvoke({ () => GetParallelData(2), () => GetParallelData(3)}),
    p = List.ParallelInvoke(lf, 10),
    r = Table.Combine(p)
    in
    r;

    GetParallelData = (postNumber as number) as table =>
    let
    Source = Json.Document(Web.Contents("https://jsonplaceholder.typicode.com/posts/" & Number.ToText(postNumber), [Headers=[Authorization="Bearer " & "asd"], ManualStatusHandling={400, 401, 403, 404, 429, 500, 503}])),
    table = Record.ToTable(Source),
    pivot = Table.Pivot(table, List.Distinct(table[Name]), "Name", "Value"),
    r = pivot
    in
    r;

    NavTable = () as table =>
    let
    source = #table({"Name", "Data", "ItemKind", "ItemName", "IsLeaf"}, {
    { "parallelRequest", parallelRequest(), "Table", "Table", true }
    }),
    navTable = Table.ToNavigationTable(source, {"Name"}, "Name", "Data", "ItemKind", "ItemName", "IsLeaf")
    in
    navTable;

     

    1. Please write example, or correct our code...how to implement? artemus 

    2. Is there other methods in Power Query, for parallelizm? artemus 

    • artemus's avatar
      artemus
      Microsoft Employee
      6 years ago

      I think it *might* work (I haven't tried this myself) if you change a line from:

      p2 = List.ParallelInvoke({ () => GetParallelData(2), () => GetParallelData(3)}),

      to

      p2 = List.ParallelInvoke([f = { () => GetParallelData(2), () => GetParallelData(3)}][f]),

       

      2. Power query will generally execute items in parellel if they can be lazly loaded. Values in Records and Tables are loaded lazly.  Also note that running your connector in Visual studio works a bit differently than in Power Bi, in order to make debugging easier. If you aren't doing so already, test your connector in Power Bi (Power Bi will reload your connector every time it is modified).

      • Anonymous's avatar
        Anonymous
        Not applicable
        6 years ago

        I tried you code, it does not work, when I pass parameters in shared:
        shared ParallelInvoke.Contents = Value.ReplaceType(ParallelInvokeImpl, ParallelInvokeType);

        or  shared ParallelInvoke.Contents = (a as text) => NavTable() as table; 

         

        It works for:

        For shared ParallelInvoke.Contents = () => NavTable() as table  ,  when it without parameter.

        artemus What can be the reason?