Forum Discussion

Txtcher's avatar
Txtcher
Helper V
1 year ago
Solved

Do Aggregations and Merges Really Take This Long in Power Query??

We do not have licenses to share reports in Power BI. We use Power Query in Excel.

I have a table loaded in the editor. Its source is a local MS Access database. There are 63,348 rows in the table. In Power Query, I deleted all columns except the ID column and a totals column. Then I grouped and summed up the totals. It sits there and churns away. I think I have waited on it now over 15 minutes.

Is something wrong with my desktop? I have no other applications open.

It is a Dell OptiPlex 

Processor Intel(R) Core(TM) i7-10700 CPU @ 2.90GHz, 2904 Mhz, 8 Core(s), 16 Logical Processor(s)

64 GB RAM.

It is my work networked computer. Unless IT is doing something behind the scenes, I just can't believe power query is this slow.

Any thoughts?

7 Replies

  • It's almost certainly an issue with how the query against your database is performing*. As a way to troubleshoot, perhaps first just load in all your rows into the model and see how that performs.

     

    I would assume the grouping should fold to the database and perform pretty well, so perhaps you are doing some transformation first that interferes with the folding?

     

    An alternative approach to doing the grouping that might work: load in all rows, then do the grouping you want with what this SQLBI article refers to as a "Reverse Linked Table"

     

    *To showcase this, here is a query that generates a table of 1M IDs and for each ID generates 5 to 10 rows of random numbers (so, depending on rng, 5-10M row table), then groups on ID and sums the random numbers. On my machine (lower specs on cpu and ram than yours), it takes ~17 seconds to load the 1M grouped rows in PBI Desktop.

     

    let
        Source = Table.FromRows(
            List.Generate(
                () => 1,
                each _ <= 1000000,
                each _ + 1,
                each {_, List.Random(Number.RoundDown(Number.RandomBetween(5, 10.9999)))}
            ),
            type table [Id = Int64.Type, Numbers = {number}]
        ),
        #"Expanded Numbers" = Table.ExpandListColumn(Source, "Numbers"),
        #"Grouped Rows" = Table.Group(
            #"Expanded Numbers", {"Id"}, {{"Sum Numbers", each List.Sum([Numbers]), type nullable number}}
        )
    in
        #"Grouped Rows"

     

  • If the ID column serves as an ID column then you probably have high cardinality, so the grouping will be a bit ineffective.

     

    Try using Table.Buffer in your original source step.  Power Query by default runs on disk/stream, but the Buffer functions force the data into memory.

    • Txtcher's avatar
      Txtcher
      Helper V

      Okay dumb question: I can't get the syntax to work:

      Thank you in advance.

  • in

    BufferMyTable

     

    Access queries don't fold into Access SQL, as you might surmise by the File.Contents step. Also, are you grouping on a unique ID column? Not sure why you would be doing that. Also, Access is just not that fast as a query source. It really sucks.

     

    --Nate