matrix dataview metadata column aggregates disappearing

My custom visual is receiving the "aggregates" field for column metadata when I filter down to a fewer number of rows, but when I send all my data, the aggregates field is not present.  Any ideas or a fix?

Thanks.

John