DAX - remove filter and add it again after full aggregation

Hello

I have a DAX question about remove filters:

This is my measure:

_AverageMachineHoursV1 = 
CALCULATE ( 
    AVERAGEX (
        FILTER (
            DIM_Date,
            YEAR ( [Date] )
                = YEAR ( [_SelectedDate] ) - 1
                && NOT ( [DayOfWeekNumber] IN { 6, 7 } )
        ),
        CALCULATE ( SUM ( FACT_Productivity[MHOURS] ) )
    ),
    REMOVEFILTERS ( DIM_Machine ),
    REMOVEFILTERS ( DIM_Department )
)

With this measure I just want a sum of the total machine hours of last year ignoring the filters on DIM_Department and DIM_Machine. I just want the average of machines hours filtered on weeknumber.

What I want is that after the result I want to keep the aggregation result but when I select a department, it only shows the machines that are related to the selected department. Like in the image below, I selected department 'B' and I expect is that machine "0000B" is removed from table but still keeping the average results. Is that possible? Even with my data model?

Thank you
Kind regards