Custom Filters for Tooltip Hover

Hi,

I have a column chart that displays spending for a set of work weeks. I'm trying to create a tool tip that creates a waterfall chart for the main changes in spending by category between the week I'm hovering over and the prior week (basically recreating what power bi does when you right click and select "explain this increase/decrease", but in a tooltip for the categories I want). 

The problem is that the hover is filtering my data for the week I'm hovering on, and then my tooltip only has one week in the filter (instead of 1week and the prior week). If I uncheck keep all filters on the tooltip page, then it doesn't use the hover filters at all. Is there a way to set so the hover filter includes prior week as well or a different solution to create this?

Thanks.