topic Re: Duration in Pipeline way longer than duration in dataflow in Pipelines

Duration in Pipeline way longer than duration in dataflow

tdenbow — Thu, 09 Nov 2023 01:17:14 GMT

I have a fabric pipeline that calls several gen2 dataflows. The refresh duration of a step in the pipeline is usually significantly higher than the refresh duration of the dataflow. For example, the pipeline will say the step took 9 minutes, but when i look at the refresh history of the dataflow it will say 4 minutes. This is nearly doubling the amount of time it takes for the pipelines to finish. Any idea why and if it is fixable?

Re: Duration in Pipeline way longer than duration in dataflow

v-cboorla-msft — Thu, 09 Nov 2023 11:21:59 GMT

Hi tdenbow

Thanks for the ask and using the Fabric Community.

At this time, we are reaching out to the internal team to get some help on this. We will update you once we hear back from them.

Appreciate your patience.
Thanks

Re: Duration in Pipeline way longer than duration in dataflow

v-cboorla-msft — Thu, 09 Nov 2023 15:11:02 GMT

Hi tdenbow

The internal team has updated that, a bug needs to be created for this issue. The team has created a bug for the same and working on it to fix it ASAP.

I will keep you posted regarding the details.

Appreciate your patience.

Thanks.

Re: Duration in Pipeline way longer than duration in dataflow

KwabenaO-A — Mon, 31 Mar 2025 13:56:39 GMT

Hi a colleague of mine has the exact issue. Has this issue been fixed now?