Forum Discussion
Table.Profile applied to each Table .Group (GroupKind,.Local & SQL set ordered): too much RAM
What is the actual question you are trying to answer? Some sort of a correlation analysis?
650 columns will be slow in any scenario. Can you please explain what your grouping criteria are, or maybe give an example?
lbendlin thanks for your message. i am trying to just get the summary of the data (min, max, etc) using the native function table.profile. i customize table.profile with BlankCount but nothing more.
i am trying to profile a large flat file. it is grouped by company name, typeofdata (revenue, etc) & numericid of the report. i have indexed the id & i sort it by the id in Source. the number of companies are dynamic. i could remove the company name & typeofreport as a test but i don't think the grouping is the holdup. i need csv's basically.
here is the m query now:
let
Source = Sql.Database("servername", "database", [Query="select * from data order by id asc"]),
#"Grouped Rows" = Table.Group(Source, {"companyname", "type", "id"}, {{"dt", each _, type table [..,...,]}}, GroupKind.Local),
profile = Table.TransformColumns(#"Grouped Rows", {"dt", each Table.Profile(_, {
{
"BlankCount", each Type.Is(_, type nullable any), each List.Count(List.Select(_, each _ = ""))
}
}
)}),
expandedprofile = Table.ExpandTableColumn(profile, "dt", {"Column", "Min", "Max", "Average", "StandardDeviation", "Count", "NullCount", "DistinctCount", "BlankCount"}, {"Column", "Min", "Max", "Average", "StandardDeviation", "Count", "NullCount", "DistinctCount", "BlankCount"})
in
expandedprofile