Handling large tables

I have simple yet large table (10+ million  rows), with the following fields

User, HoursWorked, Date , ProjectNumber, HourCode

My report should something like this:

User chooses a year  (after this filter is applied it should leave around 500K rows)

A summary table which shows users and hours worked per month for that year (grouped by ProjectNumber)

A summary table which shows users and hours worked per month for that year (grouped by HourCode)

A detail table which shows the individual hours per day per hourcode per project if they click a user from one of the summary tables

This is all very simple I can make the report no problem if I use a small table with just a subset of the large one.

But how do I make this work with the large table, both  directquery and  import table take an unacceptable long time to load

Because it seems to load the entire 10m rows and then apply the year filter