Forum Discussion

amolt's avatar
amolt
Icon for Resolver I rankResolver I
2 years ago

Dataflow runs for hours

Hello, I regularly have Gen2 dataflows that get stuck in refresh mode for hours on end, consuming all the resources of my capacity. Under normal circumstances, these dataflows only last a few minute...
Dataflows
Need Help
amolt's avatar
amolt
Icon for Resolver I rankResolver I
2 years ago

Hello HimanshuS-msft ,
Thank you for your reply.
No, I don't have any other treatment in parallel. During this last execution, the dataflow lasted more than 5h against a few minutes usually.

 

Thanks

amolt's avatar
amolt
Icon for Resolver I rankResolver I
2 years ago

Hi DataPne , Joshrodgers123 ,

Do you use a pipeline to execute your dataflows ?
In my case, I noticed that my problems came when I used a pipeline to schedule the execution of the gen2 dataflow.

  • DataPne's avatar
    DataPne
    Frequent Visitor
    2 years ago

    Hi amolt ,

    The issue seems to present itself even just when executing the pipelines manually from the UI.

    It will maybe run successfully 1/10 times. 

    Have you managed to find any work arounds on your side?

    • amolt's avatar
      amolt
      Icon for Resolver I rankResolver I
      2 years ago

      Hi DataPne ,

      I've found no other solution than to dispense with pipelines to run dataflows. I make individual schedules for each dataflow. It's really not great.
      If anyone has a better solution, I'm interested.