Forum Discussion

amolt's avatar
amolt
Icon for Resolver I rankResolver I
2 years ago

Dataflow runs for hours

Hello, I regularly have Gen2 dataflows that get stuck in refresh mode for hours on end, consuming all the resources of my capacity. Under normal circumstances, these dataflows only last a few minute...
Dataflows
Need Help
amolt's avatar
amolt
Icon for Resolver I rankResolver I
2 years ago

Hi DataPne , Joshrodgers123 ,

Do you use a pipeline to execute your dataflows ?
In my case, I noticed that my problems came when I used a pipeline to schedule the execution of the gen2 dataflow.

DataPne's avatar
DataPne
Frequent Visitor
2 years ago

Hi amolt ,

The issue seems to present itself even just when executing the pipelines manually from the UI.

It will maybe run successfully 1/10 times. 

Have you managed to find any work arounds on your side?

  • amolt's avatar
    amolt
    Icon for Resolver I rankResolver I
    2 years ago

    Hi DataPne ,

    I've found no other solution than to dispense with pipelines to run dataflows. I make individual schedules for each dataflow. It's really not great.
    If anyone has a better solution, I'm interested.