Clustered column x axis sort by with multiple value fields

Hi, 

I have a clustered column chart with a field on the x axis with 2 possible values which I want to sort in a consistent order. I have created a numeric sortby column and this works fine if there is only one value on the chart.

However, if I add additional fields as values then the categories on the x axis are appear to be sorted on the basis of size...

Is there any way around this please?