Forum Discussion
Gen2 Dataflow to Warehouse - really slow performance
Hi Scott,
It's difficult to tell what could be the reason for this performance. It could be the gateway or the computer running the gateway. The data source having some bottlenecks or perhaps something on the Fabric service end (perhaps your capacity or something different).
The best way to troubleshoot this would be to raise a support ticket and have an engineer take a closer look at this performance.
Hey Miguel,
I'm experiencing the same issues as Scott - expect that 9/10 times our dataflows are failing. Support has not been able to help. The only suggestion they had so far was to downgrade to a gateway from August..
We are looking to purchase a fabric capacity, but most of our data is on-prem. We've been trying for months to get things up and running with DFG2, but are running into error after error.
Any thoughts on what our next steps should be?
Will do - thanks miguel . Just so odd that this takes seconds (which is what I expect) in a Lakehouse but then drags out so long in a warehouse. I'll get an easily reproducible case ready, will update this thread if I'm able to narrow down the issue with support.
Thanks!
Scott
I'm experiencing the exact same thing as you. My DFG2 takes a long time to write to the warehouse, but only takes a fraction of the time to write to the lakehouse. Hopefully support has a solution for you, but I modified my process to use a Pipeline instead:
- Refresh the dataflow (destination: lakehouse)
- SQL Script to delete the data from the warehouse (beneficial for incremental data loads)
- Copy data from the lakehouse to the warehouse.
This process is much quicker than when I was trying to write directly to the warehouse.
I've spoken with support and sent them the relevant info, I'll update this post as soon as we figure out what's going on.
Thanks!
Scott
