Skip to main content
cancel
Showing results for 
Search instead for 
Did you mean: 

The ultimate Microsoft Fabric, Power BI, Azure AI & SQL learning event! Join us in Las Vegas from March 26-28, 2024. Use code MSCUST for a $100 discount. Register Now

Reply
Joshrodgers123
Advocate III
Advocate III

Dataflow Activity in Pipeline

I've been working on a pipeline that needs to run dataflows. I am noticing that the duration of the dataflow activity often far exceeds that actual runtime of the dataflow. For example, the below dataflow activity shows it took 4m 3s. When I look at the actual refresh history of the dataflow it shows it took 34s. 

 

This adds a huge amount of time to the pipeline where it is essentially doing nothing. Anyone else notice this?

 

Joshrodgers123_0-1701795833002.png

 

Joshrodgers123_1-1701795874703.png

 

5 REPLIES 5
Joshrodgers123
Advocate III
Advocate III

Hello @v-gchenna-msft 

 

Yes, I am still seeing this issue. 

 

As another test, I created a dataflow that loaded 1 single record, with 1 column using the enter data feature. So there was no data source and no data destination.

Joshrodgers123_1-1702060497018.png

 

I ran this in a pipeline 5 times consecutively. You can see the timings of the dataflow and the corresponding pipeline activity below. Overall, it took almost 20 minutes to run.. 

 

Joshrodgers123_0-1702060382373.png

 

I can send the dataflow, pipelines, and activity ids if needed, but something is definetly wrong as this should have compeleted in seconds.

Hi @Joshrodgers123 ,

Apologies for the issue that you are facing.

This might require a deeper investigation from our engineering team about your workspace and the logic behind it to properly understand what might be happening. 

Please go ahead and raise a support ticket to reach our support team:

https://support.fabric.microsoft.com/support
Please provide the ticket number here as we can keep an eye on it.

Hello @Joshrodgers123 ,

We haven’t heard from you on the last response and was just checking back to see if you got a chance to create a support ticket. If yes, please provide the ticket number here as we can keep an eye on it.

v-gchenna-msft
Community Support
Community Support

Hi @Joshrodgers123 ,

Thanks for using Fabric Community.

Apologies for the issue you are facing. I cannot find any huge difference when I tried to reproduce the scenario.

Data Pipeline:

vgchennamsft_0-1701851381055.png

Data Flow:

vgchennamsft_1-1701851437625.png


It might be an intermittent issue. Do let us know if you still observe execution time difference.



Hello @Joshrodgers123 ,

We haven’t heard from you on the last response and was just checking back to see if your query was answered. Otherwise, will respond back with the more details and we will try to help .

Helpful resources

Announcements
Fabric Community Conference

Microsoft Fabric Community Conference

Join us at our first-ever Microsoft Fabric Community Conference, March 26-28, 2024 in Las Vegas with 100+ sessions by community experts and Microsoft engineering.

Fabric Career Hub

Microsoft Fabric Career Hub

Explore career paths and learn resources in Fabric.

Fabric Hack Slide Banner

Hack Together: The Microsoft Fabric AI Global Hack

Learn from experts, get hands-on experience, and win awesome prizes.

Fabric Partner Community

Microsoft Fabric Partner Community

Engage with the Fabric engineering team, hear of product updates, business opportunities, and resources in the Fabric Partner Community.