Dataflow Activity in Pipeline

[Joshrodgers123]

I've been working on a pipeline that needs to run dataflows. I am noticing that the duration of the dataflow activity often far exceeds that actual runtime of the dataflow. For example, the below dataflow activity shows it took 4m 3s. When I look at the actual refresh history of the dataflow it shows it took 34s. 

This adds a huge amount of time to the pipeline where it is essentially doing nothing. Anyone else notice this?

[Joshrodgers123]

Hello @Anonymous 

Yes, I am still seeing this issue. 

As another test, I created a dataflow that loaded 1 single record, with 1 column using the enter data feature. So there was no data source and no data destination.

I ran this in a pipeline 5 times consecutively. You can see the timings of the dataflow and the corresponding pipeline activity below. Overall, it took almost 20 minutes to run.. 

I can send the dataflow, pipelines, and activity ids if needed, but something is definetly wrong as this should have compeleted in seconds.

[Anonymous]

Hi @Joshrodgers123 ,

Apologies for the issue that you are facing.

This might require a deeper investigation from our engineering team about your workspace and the logic behind it to properly understand what might be happening. 

Please go ahead and raise a support ticket to reach our support team:

https://support.fabric.microsoft.com/support
Please provide the ticket number here as we can keep an eye on it.

[Anonymous]

Hello @Joshrodgers123 ,

We haven’t heard from you on the last response and was just checking back to see if you got a chance to create a support ticket. If yes, please provide the ticket number here as we can keep an eye on it.

[Anonymous]

Hi @Joshrodgers123 ,

Thanks for using Fabric Community.

Apologies for the issue you are facing. I cannot find any huge difference when I tried to reproduce the scenario.

Data Pipeline:

Data Flow:

It might be an intermittent issue. Do let us know if you still observe execution time difference.

[Anonymous]

Hello @Joshrodgers123 ,

We haven’t heard from you on the last response and was just checking back to see if your query was answered. Otherwise, will respond back with the more details and we will try to help .