topic Re: Dataflow Gen2 Failures Lead to Capacity Overuse in Dataflow

Dataflow Gen2 Failures Lead to Capacity Overuse

naanii — Wed, 03 Jul 2024 06:57:56 GMT

Hi Community,

I'm often experiencing issues with my Dataflow Gen2 objects. It frequently happens that scheduled dataflows fail, causing the refresh time to extend from the usual 5 minutes to 2 hours. This results in a massive overuse of our F2 capacity. Normally, F2 is sufficient for our needs, but we constantly have to upgrade/downgrade it to keep using Fabric.

I've tried placing the dataflows in a pipeline and adding a timeout + retry mechanism. Unfortunately, once a dataflow starts, it keeps running, and the pipeline cannot cancel it.

Moreover, I can't seem to find the reason why the dataflow fails in the first place.

Do you have any ideas on how to tackle this issue effectively?

Re: Dataflow Gen2 Failures Lead to Capacity Overuse

Gmeza — Wed, 03 Jul 2024 11:39:17 GMT

This has also happened to me—I've had to stop the automatic refresh schedule completely and debug the flow before resuming refreshes. It's business-breaking since once over capacity, you can't access any Fabric items. It's kind of insane.

What is the error message after the dataflow fails?

Re: Dataflow Gen2 Failures Lead to Capacity Overuse

miguel — Wed, 03 Jul 2024 14:11:53 GMT

You can actually cancel a refresh of a dataflow

https://learn.microsoft.com/en-us/fabric/data-factory/dataflow-gen2-refresh

Re: Dataflow Gen2 Failures Lead to Capacity Overuse

naanii — Wed, 03 Jul 2024 14:50:06 GMT

Yes, I know that the dataflow can be canceled. Our dataflow runs every night. Even when it runs during office hours, we wouldn't notice since they are automated jobs.

Re: Dataflow Gen2 Failures Lead to Capacity Overuse

miguel — Mon, 08 Jul 2024 18:21:30 GMT

Definitely reach out to the support team to figure out why the Dataflow Gen2 is having issues refreshing:

https://support.fabric.microsoft.com/support