Conditional formatting on Y axis labels in bar charts

[confidentnwrong]

Hello !

I am using a bar chart and I want to change the color of the labels based on a different column. Here is a screenshot to help you visualize.

I would like the colors of the bar chart on the right to correspond to the colors of the one on the left. The field in the bar chart on the left is called "primaire" in my data.

Here is what I entered in the conditional formatting option.

But as you can see in the first screenshot, all the labels are purple.

Weirdly enough, if I filter by clicking on a category on the left, the text does change colors.

Does anyone have a solution to make my labels the right color from the start ?

Thank you in advance.