Forum Discussion

Scott_Powell's avatar
Scott_Powell
Advocate IV
2 years ago

Gen2 Dataflow to Warehouse - really slow performance

Hi, I'm hitting an issue with loading data from on-prem SQL server to a Fabric Warehouse using dataflows gen2. I'm only moving 34k rows (about 30 columns), and it's taking almost 13 minutes to do thi...
Tips and Tricks
coverturfLSH's avatar
coverturfLSH
Resolver II
2 years ago

I'm experiencing the exact same thing as you. My DFG2 takes a long time to write to the warehouse, but only takes a fraction of the time to write to the lakehouse. Hopefully support has a solution for you, but I modified my process to use a Pipeline instead:

  1. Refresh the dataflow (destination: lakehouse)
  2. SQL Script to delete the data from the warehouse (beneficial for incremental data loads)
  3. Copy data from the lakehouse to the warehouse.

This process is much quicker than when I was trying to write directly to the warehouse. 

Scott_Powell's avatar
Scott_Powell
Advocate IV
2 years ago

I've spoken with support and sent them the relevant info, I'll update this post as soon as we figure out what's going on.

 

Thanks!

Scott

  • DataPne's avatar
    DataPne
    Frequent Visitor
    2 years ago

    Hi Scott,

    Did support manage to assist you at all? Im having a call with them tomorrow for the same issue.

    Thanks a lot!
    Pne