topic Re: Fabric Data Pipeline copy task stuck in queued using on premise data gateway in Data Engineering

Fabric Data Pipeline copy task stuck in queued using on premise data gateway

mwhutch — Wed, 17 Jul 2024 13:36:03 GMT

I am attempting to use the on premise data gateway as a connection to copy data from an on premise SQL server and into a lakehouse within fabric.

I am using the latest July 2024 version of the gateway (3000.230.12), but keep running in to an issue where the copy task gets stuck in a queued state. The only way to resolve this has been to cancel the pipeline, manually restart the gateway and then kick off the pipeline again. CPU on the servers remains low whilst queued.

Has this issue been reported by anyone else and is there a resolution that can be implemented?

Re: Fabric Data Pipeline copy task stuck in queued using on premise data gateway

Anonymous — Thu, 18 Jul 2024 05:14:56 GMT

Hi @mwhutch ,

Yes, there are others who have reported the same issue, and I've put a link to the specific case below.

Fabric Data Pipeline - Copy activity stays in queu... - Page 3 - Microsoft Fabric Community

One user shared his solution out at this case link so you can see if it helps:

I wanted to confirm a couple things with you:

is this your first event using the gateway to configure the pipeline?

has it ever been configured successfully? If it has been configured successfully, did you change something in the configuration?

how long does the queued state last?

Glad you shared your solution out, it will be of great help to other users experiencing the same issue.

If you have any other questions please feel free to contact me.

Best Regards,
Yang
Community Support Team

If there is any post helps, then please consider Accept it as the solution to help the other members find it more quickly.
If I misunderstand your needs or you still have problems on it, please feel free to let us know. Thanks a lot!

Re: Fabric Data Pipeline copy task stuck in queued using on premise data gateway

mwhutch — Wed, 31 Jul 2024 10:16:24 GMT

Thank you for your message.

I am not using a staging area within Azure Blob Storage for this setting to be applicable, I am copying data directly from an on premise SQL server and into a lakehouse within fabric. This process sometimes runs without error and is an intermittent issue, no settings have been changed and is often fixed by restarting the gateway.

The queued state remains unless it is manually cancelled or times out as per the settings within the pipeline.