Measure Formats mixed up when using Calculation groups and Visual Cross Filtering at the same time

Hello Folks

I'll try to explain the situation

I have a report in which users need to chose measures to show on visuals. Among these measures, there are Counts, Costs, Ratios... I need to show the right measure in the right format depending on the user selection. To achieve this I use a disconnected table twhich Data populates a slicer to let the user chose which measure He wants to see on his visual. I added a dynamic measure which DAX code uses a SWITCH statement to return the right measure based on the selected value in the Slicer. 

As the measure are of different natures, I used Calculation groups to set the format of the Dynamic measure based on the value of the slicer. If the user selects a cost, I add the "$" sign, if they select a ratio, I show it as a %....

Everything seemed to work fine untill I recently noticed that when I cross filter two visuals on the same page, one using the dynamic measure and the second using a "Normal Meausre", the latter takes the format of the dynamic measure from the dynamic format string though I'm only applying the Dynamic format string on a visual level only not page level.

To make things a bit clearer, I attached a file to illustrate the issue. To reproduce the issue, you can just click any bar on the left Clustered bar visual:

https://1drv.ms/u/s!At_ySUsH4EHY-AePjCazr6OWRgMa?e=rVS54f

Have anyone ever seen this? Any thoughts please?

Thank you for you help