There are a number of ways you can down this "statistics" rabbit hole.
Median may be a better measure than average when you have a wide variance.
A custom "Box and Whisker" visual is one that I have used as well when I encounter a situation such as yours.
Yes, I'm sure there is a way to exclude "outliers", but someone with more expertise will have to help you with that.
Hope this helps.