Matrix Sort by Value Not Working

Hi,

I have a report showing data by seasons. I've attached an image below of the seasons. I've created a sorting measure to order how I'd like the seasons to appear (chronologically). So the logic is the first two numbers are year and Summer ends 1, Winter ends 2 so it will order chronologically. This works fine in a table. However if I drag this measure into a matrix visual and click sort by this, it still continues to sort the seasons alphabetically. 

I've included a screenshot of the matrix - I blanked out the data values as they are not important to the problem. 

In the matrix there 2 row fields, 1 column field and 4 values fields.

Has anyone experiences measures sorting tables correctly but not matrix visuals?