Calculating Yearly Total with two different date columns

I'm trying to show a user population over time that takes into account the users not only being added to the total, but also subtracted as they leave the group.  I have used the bulit in quick measure to make this:

Count of userID running total in Year = 
CALCULATE(
	COUNTA('allUsers'[userID]),
	FILTER(
		ALLSELECTED('allUsers'[joined].[Year]),
		ISONORAFTER('allUsers'[joined].[Year], MAX('allUsers'[joined].[Year]), DESC)
     )
)

Is there a way to add an AND to the filter?  I am trying to make it so the filter would say that for each year in the equation, count those records that have a joined date before or on that year, and a leave date that is greater than that year.

Filter(allUsers,'allUsers'[joined].[Year] <= Year && 'allUsers'[removed].[Year] >= Year

Any idea on how this could work?