Skip to main content
cancel
Showing results for 
Search instead for 
Did you mean: 

Get certified as a Fabric Data Engineer: Check your eligibility for a 50% exam voucher offer and join us for free live learning sessions to get prepared for Exam DP-700. Get started

Reply
Scott_Powell
Advocate III
Advocate III

Gen2 Dataflow to Warehouse - really slow performance

Hi, I'm hitting an issue with loading data from on-prem SQL server to a Fabric Warehouse using dataflows gen2. I'm only moving 34k rows (about 30 columns), and it's taking almost 13 minutes to do this (see image at bottom). Staging is disabled for this (these jobs literally just copy the data as is, with no transformations).

 

  • If I enable staging, I can see the staging objects load in < 30 seconds, and then it takes 12+ minutes to write from the staging warehouse to the actual real warehouse.
  • If I write the records to a Lakehouse instead of a Warehouse, the whole thing always completes in 30 - 45 seconds.

 

Is this type of performance typical for loading a Warehouse??? Can't believe it is, but the jobs are too simple to have anything messed up I think. Any ideas?

 

Thanks,

Scott

 

 

Scott_Powell_0-1697572196809.png

 

26 REPLIES 26
miguel
Community Admin
Community Admin

Hi Scott,

It's difficult to tell what could be the reason for this performance. It could be the gateway or the computer running the gateway. The data source having some bottlenecks or perhaps something on the Fabric service end (perhaps your capacity or something different).

 

The best way to troubleshoot this would be to raise a support ticket and have an engineer take a closer look at this performance.

https://support.fabric.microsoft.com/support

Hey Miguel,

 

I'm experiencing the same issues as Scott - expect that 9/10 times our dataflows are failing. Support has not been able to help. The only suggestion they had so far was to downgrade to a gateway from August..

 

We are looking to purchase a fabric capacity, but most of our data is on-prem. We've been trying for months to get things up and running with DFG2, but are running into error after error.

 

Any thoughts on what our next steps should be?

Will do - thanks @miguel . Just so odd that this takes seconds (which is what I expect) in a Lakehouse but then drags out so long in a warehouse. I'll get an easily reproducible case ready, will update this thread if I'm able to narrow down the issue with support.

 

Thanks!

Scott

I'm experiencing the exact same thing as you. My DFG2 takes a long time to write to the warehouse, but only takes a fraction of the time to write to the lakehouse. Hopefully support has a solution for you, but I modified my process to use a Pipeline instead:

  1. Refresh the dataflow (destination: lakehouse)
  2. SQL Script to delete the data from the warehouse (beneficial for incremental data loads)
  3. Copy data from the lakehouse to the warehouse.

This process is much quicker than when I was trying to write directly to the warehouse. 

I've spoken with support and sent them the relevant info, I'll update this post as soon as we figure out what's going on.

 

Thanks!

Scott

Hi Scott,

Did support manage to assist you at all? Im having a call with them tomorrow for the same issue.

Thanks a lot!
Pne

Helpful resources

Announcements
Las Vegas 2025

Join us at the Microsoft Fabric Community Conference

March 31 - April 2, 2025, in Las Vegas, Nevada. Use code MSCUST for a $150 discount!

Feb2025 Sticker Challenge

Join our Community Sticker Challenge 2025

If you love stickers, then you will definitely want to check out our Community Sticker Challenge!

JanFabricDE_carousel

Fabric Monthly Update - January 2025

Explore the power of Python Notebooks in Fabric!

Feb2025 NL Carousel

Fabric Community Update - February 2025

Find out what's new and trending in the Fabric community.