Why does incremental refresh takes longer than a full refresh ?

I have configured a dataflow pulling data from an offsite SQL server, millions of rows.
Earlier I was refreshing the entire table once daily and it took about 25 minutes, due to high CPU usage on the server, I upgraded to a Premium per user license and decided to use the incremental refresh option in the dataflows, everything seems configured correctly but now refresh takes even longer than before, 60+ minutes.

At first, I thought it was just the 1st refresh but no, all consecutive refreshes are as slow as the 1st one.

What could be causing this? Any potential solutions?