Calculated column not filtering matrix using a measure as a value when I collapse rows

Hoping this is an easy fix since I'm so close to what I want to show, but I need to finish this asap so any help is appreciated!

I currently have a measure that indexes data against a norm across weeks and I'm using this measure in the value field of a matrix.

The requirements are that I only show the latest two weeks averaged together in this matrix.

So I wrote a calculated column that surfaces true if the week is the latest two weeks.

When I apply that calculated column as a filter or slicer to the matrix and the weeks are listed out, then the filter condition works just fine and only the relevant data surfaces.

However, when I collapse the weeks and only show the top category row, it no longer averages just the two relevant weeks for some reason?!

Does anyone know why this might be happening? 

In this screen grab there is data for -4 through 0, but in the expanded unformatted table only the correct data is showing. 

If I take that table and collapse to the higher level row then it no longer averages just those two values (see the data point on the right in the same table that's been formatted):

Thank you!