Auto-exist and duplicated measure problem when using two Dim tables in a visual

Dear PBI community, 
I searched a lot about this issue on the forum and elsewhere, and I couldn't find the solution. Here is the situation. 
Let's first present the data. I created a simple fake data model to illustrate the problem I'm encountering. 

I have four tables in my model 

Three dimension tables :

• Interviewer
• Office
• Calendar

One fact table :

• Transactions

Here is the Interviewer table (dim)

Here is the Office table (dim) :

Here is the calendar table (dim) :

Here is the Transaction table (fact) :

Here is my data model :

Each Interviewer belongs to only one Office. Each interviewer have some transactions in the Transaction table.  

My basic measure is the number of responses by Interviewer. The number of response is defined as the number of unique Sample Unit ID for which we have Response Outcome = 1, which correspond to the following measure : 

Number Responses = 
CALCULATE(
    COUNTROWS(
        VALUES(Transactions[Sample unit ID])
    ),Transactions[Response Outcome] = 1
)

With this measure, I can create the simple table below that works properly :

The problem comes when I want to add the total number of response by Office in the same table. I use the following measure to do so : 

Number Responses Office = 
CALCULATE(
    [Number Responses],
    ALLSELECTED(Interviewer)
)

My problem may be in this formula, but I need to use ALLSELECTED() filter function because I need to be able to modify this measure from the external slicers. Here is the result I get when I add this measure in my table : 

As you can see, since I'm using two columns from two different dim tables, the Auto-Exact feature is disabled and I get the full cardinality of the tables Interviewer and Office (all possible combinations). The values of the measure "Number Responses Office" are accurate but it created a lot of impossible combinations of Interviewer and Office. How can I avoid this ? I don't want to manually filter out the rows where "Number Responses" = 0, because an interviewer may have no response, ans still be a valid data point. I would like to correct my measure so that it do not create all the impossible combinations of interviewers and offices. 

You can acces my PBIX fiel here : 

https://drive.google.com/file/d/1SoKd0ze7xRII27cY5vtgNDkavOaiSZID/view?usp=sharing

Please let me know if the link doesn't work. 

Any help will be much appreciated!

Thanks,