For any visual on your report (not just matrix) you can edit the relationships, and in this case turn off any filtering/highlighting that occurs.

First, select the first slicer, click on the Format button (top ribbon) and select "Edit Interactions".  You should a filter icon and possibly a pie icon and a strike through icon appear for all other visuals.  Then, on the matrix you want to have no interaction, just select the strikethrough option to disable the interaction.

Hi strangerMike,

For detail steps, you can check jpereztang's suggestion in this similar thread: Visual Interactions on Pie Chart cancelling out .

Reference:
Visualization interactions in a Power BI report

Best Regards,
Qiuyun Yu