Calculating Changes on the fly between two tables

Ok, so I was overthinking this a little bit and that has helped. The problem is, when I filter it by the item, it removes all Names that aren't related to that Item. 
So the overall averages work a treat if all items are selected.  but the problem comes when I filter it. so I do have a table with just the names in it, which has links to both the scores and the uplifts tables. so if i do all the calculations within the names table, but the filter is based on the items in uplifts, it removes any item in the names table that doesn't have that item. and then averages are just the averages of those items, not all items with those uplifts. 
for the example above, that means I'm getting: an average of 2 with no uplifts - great. 

And if I do all scores + all uplifts / count of rows = 2.5  - great 😄 

but then if i filter the visual to just show me items 1+2, that's giving me the average of (2.5+3.5)/2 = 3  whereas it should be 2.4 - it's ignoring the Names of stuff that doesn't have that mapping. and short of mapping every Item to every Name with a value of 0 which is kinda crazy and would be a very very big table, i'm not sure how to get to the average i'm looking for?