Forum Discussion
Table.Profile applied to each Table .Group (GroupKind,.Local & SQL set ordered): too much RAM
i've made the following change but no luck yet: instead of adding a column table type of table.profile, i do table.transformcolumn of the data & apply table.profile instead. i've tried switching off parallel load also but load gets stuck at 96% RAM eventually & CPU usage of container averaging around 3% only..
What is the actual question you are trying to answer? Some sort of a correlation analysis?
650 columns will be slow in any scenario. Can you please explain what your grouping criteria are, or maybe give an example?
- Anonymous4 years agoNot applicable
lbendlin thanks for your message. i am trying to just get the summary of the data (min, max, etc) using the native function table.profile. i customize table.profile with BlankCount but nothing more.
i am trying to profile a large flat file. it is grouped by company name, typeofdata (revenue, etc) & numericid of the report. i have indexed the id & i sort it by the id in Source. the number of companies are dynamic. i could remove the company name & typeofreport as a test but i don't think the grouping is the holdup. i need csv's basically.
here is the m query now:
let
Source = Sql.Database("servername", "database", [Query="select * from data order by id asc"]),#"Grouped Rows" = Table.Group(Source, {"companyname", "type", "id"}, {{"dt", each _, type table [..,...,]}}, GroupKind.Local),
profile = Table.TransformColumns(#"Grouped Rows", {"dt", each Table.Profile(_, {
{
"BlankCount", each Type.Is(_, type nullable any), each List.Count(List.Select(_, each _ = ""))
}
}
)}),
expandedprofile = Table.ExpandTableColumn(profile, "dt", {"Column", "Min", "Max", "Average", "StandardDeviation", "Count", "NullCount", "DistinctCount", "BlankCount"}, {"Column", "Min", "Max", "Average", "StandardDeviation", "Count", "NullCount", "DistinctCount", "BlankCount"})
in
expandedprofile - Anonymous4 years agoNot applicable
lbendlin i should also say that initially i had split up all the groupings manually as separate queries & the profile took about 1.5hrs. i turned off parallel & background previews. not sure if that helps. thanks. i am trying to make it dynamic though as the groupings will change frequently.
- lbendlin4 years ago
Super User
Frankly I wouldn't bother with Power Query for that. There are tools like Alteryx or Knime that are more specialized and (in the case of Alteryx) have far better performance.
Anyway - would you be able to provide a sample extract that can demonstrate the issue?