Forum Discussion

Scott_Powell's avatar
Scott_Powell
Advocate IV
2 years ago

Gen2 Dataflow to Warehouse - really slow performance

Hi, I'm hitting an issue with loading data from on-prem SQL server to a Fabric Warehouse using dataflows gen2. I'm only moving 34k rows (about 30 columns), and it's taking almost 13 minutes to do thi...
Tips and Tricks
Scott_Powell's avatar
Scott_Powell
Advocate IV
2 years ago

Will do - thanks miguel . Just so odd that this takes seconds (which is what I expect) in a Lakehouse but then drags out so long in a warehouse. I'll get an easily reproducible case ready, will update this thread if I'm able to narrow down the issue with support.

 

Thanks!

Scott

coverturfLSH's avatar
coverturfLSH
Resolver II
2 years ago

I'm experiencing the exact same thing as you. My DFG2 takes a long time to write to the warehouse, but only takes a fraction of the time to write to the lakehouse. Hopefully support has a solution for you, but I modified my process to use a Pipeline instead:

  1. Refresh the dataflow (destination: lakehouse)
  2. SQL Script to delete the data from the warehouse (beneficial for incremental data loads)
  3. Copy data from the lakehouse to the warehouse.

This process is much quicker than when I was trying to write directly to the warehouse. 

  • Scott_Powell's avatar
    Scott_Powell
    Advocate IV
    2 years ago

    I've spoken with support and sent them the relevant info, I'll update this post as soon as we figure out what's going on.

     

    Thanks!

    Scott

    • DataPne's avatar
      DataPne
      Frequent Visitor
      2 years ago

      Hi Scott,

      Did support manage to assist you at all? Im having a call with them tomorrow for the same issue.

      Thanks a lot!
      Pne