Line graph drill down

I have a graph with survey results going back to October 2019. It shows all surveys with dates. I also have a dimention table with additional columns for fiscal year, so that I can organize my slicers and graphs by fiscal year.

Here is the problem:  when  I drill down to see line graph results by month, October shows me the average of surveys from both 2019 and 2020. This will really confuse the viewer, he will think he's looking at the average for the month in current fiscal year. I tried changing the selection 'categorical' to 'continuous' in the line graph under x-axis, but it doesn't work (it just won't switch). 

Ideally, I would like to show all months (including those with no survey) in the order of when surveys were submitted, and divided by the fiscal year in which they were submitted.

Any ideas?