In fact, I think these aggregate features work as expected. The default aggregated features are used to summarize value based on category(the category fields mean axis, legends, and 'do not summarized' fields that are in the visuals).
If you add more fields to your visuals without aggregate, they obviously will affect the current aggregate, and try to change the new effect to summary value include all existing categories.
In addition, you can refer to the following links if helps:
User-defined aggregations - Power BI | Microsoft Learn
Work with aggregates (sum, average, and so on) in Power BI - Power BI | Microsoft Learn
Row Context and Filter Context in DAX - SQLBI
Community Support Team _ Xiaoxin
If this post helps, please consider accept as solution to help other members find it more quickly.