ALL() is not working as expected

Hello,

I have a  matrix visual with 8 to 9 columns and have one page level filter(Values:1,2,4--Applied filter not in 4) and one report level filter(Applied filter sales not in US) applied to the report. Now, when ther user filter(Yes/No filter name) out of 8 to 9 columns, 3 columns values should not be effected in the matrix but when the user selects other filters, values should change. I have all the columns and filters coming from the same table.

Can anyone help me out in getting this?

I have tried using this

Calculate(DISTINCTCOUNT(column),All(Yes/No)) is not giving me the correct result.