Forum Discussion

sjrrkb123's avatar
sjrrkb123
Icon for Helper III rankHelper III
2 years ago
Solved

Combining Multiple Parquet files into a table in Dataflows Very Slow

I have two dataflows pointing to the same set of data. Dataflow A is on SQL server (on prem) and dataflow B is in multiple .parquet files that need to be combined into a single table.   When I run ...
Dataflow
Need Help
  • sjrrkb123's avatar
    sjrrkb123
    2 years ago

    Turns out our best solution will be to spin up a vm, install a gateway on it, and use it for the dataflow refresh from alds gen2 datalakes.

lbendlin's avatar
lbendlin
Icon for Super User rankSuper User
2 years ago

Don't use an on-prem gateway for your Azure data sources. You may not even need a VNet gateway.

sjrrkb123's avatar
sjrrkb123
Icon for Helper III rankHelper III
2 years ago

Turns out our best solution will be to spin up a vm, install a gateway on it, and use it for the dataflow refresh from alds gen2 datalakes.