Do Aggregations and Merges Really Take This Long in Power Query??

It's almost certainly an issue with how the query against your database is performing*. As a way to troubleshoot, perhaps first just load in all your rows into the model and see how that performs.

I would assume the grouping should fold to the database and perform pretty well, so perhaps you are doing some transformation first that interferes with the folding?

An alternative approach to doing the grouping that might work: load in all rows, then do the grouping you want with what this SQLBI article refers to as a "Reverse Linked Table"

*To showcase this, here is a query that generates a table of 1M IDs and for each ID generates 5 to 10 rows of random numbers (so, depending on rng, 5-10M row table), then groups on ID and sums the random numbers. On my machine (lower specs on cpu and ram than yours), it takes ~17 seconds to load the 1M grouped rows in PBI Desktop.

let
    Source = Table.FromRows(
        List.Generate(
            () => 1,
            each _ <= 1000000,
            each _ + 1,
            each {_, List.Random(Number.RoundDown(Number.RandomBetween(5, 10.9999)))}
        ),
        type table [Id = Int64.Type, Numbers = {number}]
    ),
    #"Expanded Numbers" = Table.ExpandListColumn(Source, "Numbers"),
    #"Grouped Rows" = Table.Group(
        #"Expanded Numbers", {"Id"}, {{"Sum Numbers", each List.Sum([Numbers]), type nullable number}}
    )
in
    #"Grouped Rows"