topic Re: Dataflow executed by Pipeline runs forever in Pipelines

Dataflow executed by Pipeline runs forever

Joschko — Tue, 12 Dec 2023 08:34:58 GMT

Hey,

I am running a data flow through a pipeline.
The dataflow normally runs every 5 minutes and takes 20 seconds.
But I observe that the dataflow takes 5-8 minutes at the full and half hour.
The other runs are done within 20 seconds.
Since last night the dataflow runs now for 10 Hours.
Is there any possibility to cancel the dataflow?
Do I have a chance to make the outlier dataflows (5-8 minutes) running faster?

Thank you for reading
Jörg

Re: Dataflow executed by Pipeline runs forever

v-cboorla-msft — Tue, 12 Dec 2023 15:48:04 GMT

Hi Joschko

Thanks for using Fabric Community.

Apologies for the issue that you are facing here.

Try to upgrade the gateway version to latest, this might help me in resolving your issue.

For more details please refer Link.

I hope this helps. Please do let us know if you have any further questions or If the issue still persists.

Thanks

Re: Dataflow executed by Pipeline runs forever

Joschko — Tue, 12 Dec 2023 16:00:56 GMT

Hi,

Thank you very much for your response.

no on-premises gateway involved. The Dataflow gets the data from a REST api.

I alreday opened a ticket to cancel the dataflow.

Unfortunelty it seems there is no option to cancel a Gen2 Dataflow in Fabric.

Thank you

Jörg

Re: Dataflow executed by Pipeline runs forever

Younesselouajad — Tue, 12 Dec 2023 20:11:24 GMT

Hi,
Since yesterday, I also have a dataflow gen2 running forever. I use a pipeline to schdule the refresh. All my dataflows have been succuffly refreshing excpet this one that pull data from a REST API.
Hope there is a solution.

Re: Dataflow executed by Pipeline runs forever

Joschko — Wed, 13 Dec 2023 08:40:39 GMT

Hi,

ok. The dataflow timed out after 24 hours.

Still do not know why the dataflow hang.

A cancel option would be great.

Thanks

Re: Dataflow executed by Pipeline runs forever

v-cboorla-msft — Fri, 22 Dec 2023 05:29:25 GMT

Hi Joschko

Following up to see whether your issue got resolved or still persists.
In case if you have any resolution please do share that same with the community as it can be helpful to others.
Otherwise, will respond back with the more details and we will try to help.

Thanks

Re: Dataflow executed by Pipeline runs forever

v-cboorla-msft — Tue, 26 Dec 2023 07:20:55 GMT

Hi Joschko

We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet.
In case if you have any resolution please do share that same with the community as it can be helpful to others.
If you have any question relating to the current thread, please do let us know and we will try out best to help you.
In case if you have any other question on a different issue, we request you to open a new thread.

Thanks

Re: Dataflow executed by Pipeline runs forever

Joschko — Wed, 27 Dec 2023 08:32:05 GMT

Hi,

I restarted the pipeline and unfortunelty the problem still exists.

The Duration is not predictable.

Thank you for your help

Re: Dataflow executed by Pipeline runs forever

Joshrodgers123 — Wed, 27 Dec 2023 20:39:20 GMT

I am seeing the same issues as well. I have a dataflow that normally runs for less than 20 seconds. When run from a pipeline, it can take up to 10 minutes.