Conditional formatting breaks with cross filter

Hi, 

I have this chart using a color measure, IF([measure]<0,"red","blue") which works fine.

The issue is when the user cross filters this happens:

I cant seem to find the solution, is it possible to fix it?

Thank you