Problem with subtracting two columns in a matrix

I have a matrix table that shows the budget of investments. The matrix has 4 row categories: Location, Investment type, Description and Investment number. And the values are: Budget, Filling the budget as of Q1, Q1 Difference, Filling the budget as of Q2, Q2 Difference and so on..

The matrix should look something like this:

and I do get this result without the "Q1 Difference" calculation, but when I enter the calculation I get this:

As you can see it displays all the investment numbers (even when they don't apply to a certain Description, Investment type or location) under every description and the budget number multiplies behind all the investments. Only the "Filling the budget as of Q1" is behind the correct row.

I've tried with a regular Quick Measure, I've tried with =CALCULATE(SUM([Budget])) - CALCULATE(SUM([Filling the budget as of Q1])), which worked a couple of days ago, but then I had other things wrong with my Report and I've changed the queries since then quite a bit. Also tried doing the calculation with SQL, which works behind all the rows, but the grand total number will be off.

Been stuck on this problem for a while now, any help is appreciated.