Refresh timeouts - No load on Gateway or SQL DB

The primary error message is:
"Before the data import for finished, its data source timed out. Double-check whether that data source can process import queries, and if it can, try again." and sometimes we also get the "Power BI may have lost connectivity with the data gateway during this dataset refresh. Please make sure that the data gateway is up and running for the next scheduled refresh." error

I am confident that its not a problem with the SQL database as there is barely any activity on it. At most 2 queries are running simultaneously and as stated in my original post, almost no CPU load. Each individual dataset should easily have 10 simultaneously executing queries.
And yes, I did trigger the refreshes on different datasets for this test.
I should also point out that installing the gateway on a less powerful server actually solves the issue and is our current workaround. I'm also in contact with the support regarding this issue.