Forum Discussion

Joshrodgers123's avatar
Joshrodgers123
Advocate V
2 years ago

Dataflow Activity in Pipeline

I've been working on a pipeline that needs to run dataflows. I am noticing that the duration of the dataflow activity often far exceeds that actual runtime of the dataflow. For example, the below dataflow activity shows it took 4m 3s. When I look at the actual refresh history of the dataflow it shows it took 34s. 

 

This adds a huge amount of time to the pipeline where it is essentially doing nothing. Anyone else notice this?

 

 

 

pipelines

5 Replies

  • Anonymous's avatar
    Anonymous
    Not applicable
    2 years ago

    Hi Joshrodgers123 ,

    Thanks for using Fabric Community.

    Apologies for the issue you are facing. I cannot find any huge difference when I tried to reproduce the scenario.

    Data Pipeline:

    Data Flow:


    It might be an intermittent issue. Do let us know if you still observe execution time difference.



    • Anonymous's avatar
      Anonymous
      Not applicable
      2 years ago

      Hello Joshrodgers123 ,

      We haven’t heard from you on the last response and was just checking back to see if your query was answered. Otherwise, will respond back with the more details and we will try to help .

  • Joshrodgers123's avatar
    Joshrodgers123
    Advocate V
    2 years ago

    Hello Anonymous 

     

    Yes, I am still seeing this issue. 

     

    As another test, I created a dataflow that loaded 1 single record, with 1 column using the enter data feature. So there was no data source and no data destination.

     

    I ran this in a pipeline 5 times consecutively. You can see the timings of the dataflow and the corresponding pipeline activity below. Overall, it took almost 20 minutes to run.. 

     

     

    I can send the dataflow, pipelines, and activity ids if needed, but something is definetly wrong as this should have compeleted in seconds.

    • Anonymous's avatar
      Anonymous
      Not applicable
      2 years ago

      Hi Joshrodgers123 ,

      Apologies for the issue that you are facing.

      This might require a deeper investigation from our engineering team about your workspace and the logic behind it to properly understand what might be happening. 

      Please go ahead and raise a support ticket to reach our support team:

      https://support.fabric.microsoft.com/support
      Please provide the ticket number here as we can keep an eye on it.

      • Anonymous's avatar
        Anonymous
        Not applicable
        2 years ago

        Hello Joshrodgers123 ,

        We haven’t heard from you on the last response and was just checking back to see if you got a chance to create a support ticket. If yes, please provide the ticket number here as we can keep an eye on it.