How to ensure small multiple only shows data with non-zero data.

I am trying to set up a clustered bar chart on power bi. Currently I have managers on the y-axis and hours provided on the x-axis. I also have a bucketing method with small multiples for different teams. Some managers are on several teams and their hours are reflected respectively in each small multiple. However, a lot of small multiples only have a few managers to that particular team, but all of the other manager names are still on the y-axis with lots of blank columns. How do i make it that each small multiple only shows managers/hours that have hours for that team and not show all of the other managers.

Team 1 for example might have managers A, B, C. Team 2 might have managers B, C. On team 2's small multiple I only want to show manager B and C's hours for those teams. Team 1 I want to show managers A, B, C's hours for that team. 

I've tried using measures, changing the x/y-axis sources, visual level filters, but none of them worked. 
Visible Hours =
VAR Hours = SUM(FactTable[Hours])
RETURN IF(Hours > 0, Hours)

Below is a screenshot of my issue, I am getting a lot of blank space shown by the highlighted area that I would like to cut out for each specific small multiple/team.