Forum Discussion

Scott_Powell's avatar
Scott_Powell
Icon for Advocate IV rankAdvocate IV
2 years ago

Gen2 Dataflow to Warehouse - really slow performance

Hi, I'm hitting an issue with loading data from on-prem SQL server to a Fabric Warehouse using dataflows gen2. I'm only moving 34k rows (about 30 columns), and it's taking almost 13 minutes to do thi...
Tips and Tricks
Scott_Powell's avatar
Scott_Powell
Icon for Advocate IV rankAdvocate IV
2 years ago

Will do - thanks miguel . Just so odd that this takes seconds (which is what I expect) in a Lakehouse but then drags out so long in a warehouse. I'll get an easily reproducible case ready, will update this thread if I'm able to narrow down the issue with support.

 

Thanks!

Scott

coverturfLSH's avatar
coverturfLSH
Icon for Resolver II rankResolver II
2 years ago

I'm experiencing the exact same thing as you. My DFG2 takes a long time to write to the warehouse, but only takes a fraction of the time to write to the lakehouse. Hopefully support has a solution for you, but I modified my process to use a Pipeline instead:

  1. Refresh the dataflow (destination: lakehouse)
  2. SQL Script to delete the data from the warehouse (beneficial for incremental data loads)
  3. Copy data from the lakehouse to the warehouse.

This process is much quicker than when I was trying to write directly to the warehouse.