Need help understanding extra queries generated by enabling RLS

I'm very new to power bi so I'm hopefully just missing something obvious but here we go...

I'm developing a report where I get data from a single SQL Server database, some tables are imported and some are DirectQuery (BIG fact tables, import is not an option).

I've got one fact table which (among other things) contains the columns system, device, date, timestamp, event and event_value. I have a line graph where I put the timestamp on the x-axis and the event_value on the y-axis and I have single select slicers for the system, device and date columns. This works fine when I don't enable row level security. The queries sent to the database contains the correct filters and get good execution plans.

If I enable row level security, an extra query is sent to the database which (to me) makes no sense. The query is fetching all distinct values from the timestamp column and it has no filters except one on the system column which comes from the RLS rule so it is querying the table from the beginning of time until now. Below is an edited version of the query that is sent:

SELECT
TOP (1000001) [t16].[timestamp]
FROM
(
(
select [$Table].[system] as [system],
[$Table].[device] as [device],
[$Table].[date] as [date],
[$Table].[timestamp] as [timestamp],
[$Table].[event] as [event],
[$Table].[event_value] as [event_value],
from [fact].[dv] as [$Table]
)
)
AS [t16]
WHERE
(
([t16].[system] IN (15,30,16,31,17,1,18,33,2,34,3,19,35,20,36,4,21,37,5,22,6,23,7,24,9,11,25,26,12,27,13,29))
)
GROUP BY [t16].[timestamp]

The RLS rule is on another table "access_list" which contains two columns, user and system, there are multiple rows per terminal and multiple rows per user. The rule is 'admin access_list'[user] = USERPRINCIPALNAME(), this table is in turn linked via the system column to the dimension table "system" (imported) which in turn is linked to the fact table.

I've tried to have to have the access_list as DirectQuery, imported and dual, I've tried to connect it directly to the fact table, nothing makes a difference.

I've seen that the same thing happens for table visuals where I show a column from the fact table which isn't aggregated, for example if I have a table with the columns system, device, event and sum(event_value), a query runs that gets all distinct values for the "event" column.

I'm wondering if the cause if this is the relation between the system table and the access_list table. Since the cardinality is "many to one" from access_list to system I need to set cross filter direction to both in order for the RLS filter to propagate down to the fact table.