Microsoft Fabric Community Conference 2025, March 31 - April 2, Las Vegas, Nevada. Use code MSCUST for a $150 discount.
Register nowGet certified as a Fabric Data Engineer: Check your eligibility for a 50% exam voucher offer and join us for free live learning sessions to get prepared for Exam DP-700. Get started
Hi, I'm hitting an issue with loading data from on-prem SQL server to a Fabric Warehouse using dataflows gen2. I'm only moving 34k rows (about 30 columns), and it's taking almost 13 minutes to do this (see image at bottom). Staging is disabled for this (these jobs literally just copy the data as is, with no transformations).
Is this type of performance typical for loading a Warehouse??? Can't believe it is, but the jobs are too simple to have anything messed up I think. Any ideas?
Thanks,
Scott
Hi Scott,
It's difficult to tell what could be the reason for this performance. It could be the gateway or the computer running the gateway. The data source having some bottlenecks or perhaps something on the Fabric service end (perhaps your capacity or something different).
The best way to troubleshoot this would be to raise a support ticket and have an engineer take a closer look at this performance.
Hey Miguel,
I'm experiencing the same issues as Scott - expect that 9/10 times our dataflows are failing. Support has not been able to help. The only suggestion they had so far was to downgrade to a gateway from August..
We are looking to purchase a fabric capacity, but most of our data is on-prem. We've been trying for months to get things up and running with DFG2, but are running into error after error.
Any thoughts on what our next steps should be?
Will do - thanks @miguel . Just so odd that this takes seconds (which is what I expect) in a Lakehouse but then drags out so long in a warehouse. I'll get an easily reproducible case ready, will update this thread if I'm able to narrow down the issue with support.
Thanks!
Scott
I'm experiencing the exact same thing as you. My DFG2 takes a long time to write to the warehouse, but only takes a fraction of the time to write to the lakehouse. Hopefully support has a solution for you, but I modified my process to use a Pipeline instead:
This process is much quicker than when I was trying to write directly to the warehouse.
I've spoken with support and sent them the relevant info, I'll update this post as soon as we figure out what's going on.
Thanks!
Scott
Hi Scott,
Did support manage to assist you at all? Im having a call with them tomorrow for the same issue.
Thanks a lot!
Pne
March 31 - April 2, 2025, in Las Vegas, Nevada. Use code MSCUST for a $150 discount!
If you love stickers, then you will definitely want to check out our Community Sticker Challenge!
User | Count |
---|---|
3 | |
1 | |
1 | |
1 | |
1 |