Fabric Semantic Model Vs SSAS Tabular Cube

If your SSAS was a "standard import mode model" you're doing a huge jump moving to Fabric DL.

In import mode you've all the data processed and compressed at processing time and then you mount reports in live connection on top of if it.

in Fabric Direct Lake, data resides in Warehouses or Lakehouses as Delta Tables (Parquet files) and the needed data are paged into memory when requested by users. Columns have a temperature and Fabric can decide to page them out. the time to load data into memory affects the query/report time. 
according to the capacity size you can experience also performance degradation when the engine is not able to use Direct Lake and switch to Direct Query over SQL endpoint (this process is know as Fallback Direct Lake overview - Microsoft Fabric | Microsoft Learn). 
When fallback happens you're basically going to the slowest storage mode automatically and your queries may be affected.

I strongly suggest to switch to Import if you have no real reasons to use Direct Lake. If your capacity is correctly sized your performance would be similar to SSAS.

I suggest to double check your capacity is not overloaded that would cause queries (interactive operations) to be throttled even if DL is happening.

I suggest to disable auto fallback if you want to stay direct lake to clearly catch what is happening behind the scenes.