Dataflow Gen2 Failures Lead to Capacity Overuse

[naanii]

Hi Community,

I'm often experiencing issues with my Dataflow Gen2 objects. It frequently happens that scheduled dataflows fail, causing the refresh time to extend from the usual 5 minutes to 2 hours. This results in a massive overuse of our F2 capacity. Normally, F2 is sufficient for our needs, but we constantly have to upgrade/downgrade it to keep using Fabric.

I've tried placing the dataflows in a pipeline and adding a timeout + retry mechanism. Unfortunately, once a dataflow starts, it keeps running, and the pipeline cannot cancel it.

Moreover, I can't seem to find the reason why the dataflow fails in the first place.

Do you have any ideas on how to tackle this issue effectively?

[miguel]

You can actually cancel a refresh of a dataflow

https://learn.microsoft.com/en-us/fabric/data-factory/dataflow-gen2-refresh

[naanii]

Yes, I know that the dataflow can be canceled. Our dataflow runs every night. Even when it runs during office hours, we wouldn't notice since they are automated jobs.

[miguel]

Definitely reach out to the support team to figure out why the Dataflow Gen2 is having issues refreshing:

https://support.fabric.microsoft.com/support

[Gmeza]

This has also happened to me—I've had to stop the automatic refresh schedule completely and debug the flow before resuming refreshes. It's business-breaking since once over capacity, you can't access any Fabric items. It's kind of insane.

What is the error message after the dataflow fails?