topic Re: Fabric Data Pipeline - Copy activity stays in queued status in Pipelines

Fabric Data Pipeline - Copy activity stays in queued status

Anonymous — Tue, 16 Apr 2024 17:17:00 GMT

I have a data pipeline with a copy activity that copies data from a table in sql server into Fabric - Warehouse. The copy activity stays in queued status for a very long time and never gets picked up for processing. What could be the reason? How can I troubleshoot this issue? A similar pipeline that copies data into a lakehouse works fine.

Re: Fabric Data Pipeline - Copy activity stays in queued status

v-cboorla-msft — Wed, 17 Apr 2024 12:47:15 GMT

Hi Anonymous

Thanks for using Microsoft Fabric Community.

Apologies for the issue you are facing.

In order to understand your specific data copy scenario, Could you please share some more details about your copy data activity. This will help me to understand the query and guide in better way. If possible, could you provide information on the following:

Data Transformation Specifications : Are there any transformations being applied to the data during the copy process?

Screenshots : If feasible, sharing screenshots of the source and sink configuration within your data copy activity would be highly beneficial for visualization purposes.

Thanks.

Re: Fabric Data Pipeline - Copy activity stays in queued status

Anonymous — Fri, 19 Apr 2024 14:06:02 GMT

Hello v-cboorla-msft . Adding screenshots below. I am just testing with a narrow table that has a few records. What is puzzling is that when I perform the same copy data activity that copies into a lakehouse works. The issue is with Warehouse.
Source : SQL Server installed in an Azure VM, connectivity to this from Fabric exists.
Destination: Fabric Warehouse
Data Transformations : None

Re: Fabric Data Pipeline - Copy activity stays in queued status

v-cboorla-msft — Fri, 19 Apr 2024 17:41:32 GMT

Hi Anonymous

Thank you sharing the details.

I have successfully replicated the scenario and confirmed the pipeline functioned as expected. Please find the attached screenshots.

Source:

Destination:

Mapping:

Settings:

Copy data details:

Could you please recheck the pipeline configuration, data source connections. Also under settings tab try to select workspace instead of external for Data store type and try to execute the pipeline.

If the issue still persists, please do let us know. Glad to help.

Note : If you are copying the data using cloud connection for loading the data to Warehouse, Microsoft Fabric support onelake as staging.

I hope this information helps.

Thank you.

Re: Fabric Data Pipeline - Copy activity stays in queued status

v-cboorla-msft — Mon, 22 Apr 2024 06:04:03 GMT

Hi Anonymous

We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution please do share that same with the community as it can be helpful to others.
Otherwise, will respond back with the more details and we will try to help.

Thanks.

Re: Fabric Data Pipeline - Copy activity stays in queued status

Anonymous — Mon, 22 Apr 2024 12:28:57 GMT

Hi Anonymous ,

I was facing the same issue as you. In my case (hope it helps you), after working with MS support, a new version of the on-prem data gateway was released (on April 18th 2024 - 3000.218.8). As soon as I installed this new version, the pipelines started working.

Re: Fabric Data Pipeline - Copy activity stays in queued status

Anonymous — Mon, 22 Apr 2024 13:12:29 GMT

Thanks for the follow up v-cboorla-msft . I figured out that the storage account I was using for staging was not reachable. After swtiching the storage account, the activity gets picked up for processing. I see file generated in the staging folder in the storage account. But, the activity fails with the following error when it tries to connect to the warehouse.

ErrorCode=SqlFailedToConnect,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=Cannot connect to SQL Database. Please contact SQL server team for further support. Server: 'dkjfghkdfgldgdlfgjldjfg-dfkjghkdrt543dfg9345jgfd.datawarehouse.fabric.microsoft.com', Database: '56756757-d5435-456g-5675-456456456fre', User: ''. Check the connection configuration is correct, and make sure the SQL Database firewall allows the Data Factory runtime to access.,Source=Microsoft.DataTransfer.ClientLibrary,''Type=System.Data.SqlClient.SqlException,Message=A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is configured to allow remote connections. (provider: Named Pipes Provider, error: 40 - Could not open a connection to SQL Server),Source=.Net SqlClient Data Provider,SqlErrorNumber=53,Class=20,ErrorCode=-2146232060,State=0,Errors=[{Class=20,Number=53,State=0,Message=A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is configured to allow remote connections. (provider: Named Pipes Provider, error: 40 - Could not open a connection to SQL Server),},],''Type=System.ComponentModel.Win32Exception,Message=The network path was not found,Source=,'

Re: Fabric Data Pipeline - Copy activity stays in queued status

Anonymous — Mon, 22 Apr 2024 13:15:29 GMT

Thanks Anonymous . Does the data factory that is part of fabric instance use a gateway? If so, where can I find the gateway? In my case I am copying data from a SQL Server in an Azure VM to Warehouse residing in fabric. Data pipeline I created is also part of the fabric instance.

Re: Fabric Data Pipeline - Copy activity stays in queued status

v-cboorla-msft — Mon, 22 Apr 2024 13:18:19 GMT

Hi Anonymous

Thanks Anonymous for sharing the details.

For the latest version of on-prem gateway please refer : On-premises data gateway

I hope this helps.

Thank you.

Re: Fabric Data Pipeline - Copy activity stays in queued status

Anonymous — Mon, 22 Apr 2024 13:18:58 GMT

My appologies. I read your post too quickly and assumed that you were having the same issue as me. In my case, this issue was only happenning when I was copying data from my on-prem SQL server (in a VPC on AWS). Very sorry to have mis-lead you.

Re: Fabric Data Pipeline - Copy activity stays in queued status

Anonymous — Mon, 22 Apr 2024 13:33:14 GMT

Thanks for following up v-cboorla-msft . After switching the storage account to another one, the copy activity started processing, data is copied to the staging folder in the storage account. But, when it tries to connect to the warehouse to copy the data, it fails with the following error.

Re: Fabric Data Pipeline - Copy activity stays in queued status

v-cboorla-msft — Mon, 22 Apr 2024 13:35:12 GMT

Hi Anonymous

In the Settings menu, under Data Store Type, could you please change the selection to "Workspace" instead of "External". After doing the changes we recommend re-running the pipeline, this might help you in resolving your issue.

If the issue still persists, please do let us know, we will try to help.

Note : If you are copying the data using cloud connection for loading the data to Warehouse, Microsoft Fabric support onelake as staging.

Hope this helps.

Thank you.

Re: Fabric Data Pipeline - Copy activity stays in queued status

Anonymous — Mon, 22 Apr 2024 13:39:31 GMT

I get validation error if I switch to 'workspace'.

Re: Fabric Data Pipeline - Copy activity stays in queued status

v-cboorla-msft — Mon, 22 Apr 2024 13:50:10 GMT

Hi Anonymous

Apologies for the issue that you facing here.

Could you please share the details of your connection settings that you are uisng. This will help me to guide you in better way.

Thank you.

Re: Fabric Data Pipeline - Copy activity stays in queued status

Anonymous — Mon, 22 Apr 2024 14:04:09 GMT

v-cboorla-msft The warehouse and pipeline are part of the workspace. The copy activity does not ask for any connection string, see screenshots below. I execute the pipeline by clicking the Run button on the pipeline screen. I have logged into https://app.fabric.microsoft.com/ as me using my company account.

Re: Fabric Data Pipeline - Copy activity stays in queued status

v-cboorla-msft — Mon, 22 Apr 2024 14:25:28 GMT

Hi Anonymous

Thank you for sharing the details. But could you please try to send the below details.

Thank you.

Re: Fabric Data Pipeline - Copy activity stays in queued status

Anonymous — Mon, 22 Apr 2024 14:37:10 GMT

v-cboorla-msft Are you saying that I should use 'SQL Server' as my destination instead of 'Warehouse'? Even if I try to do that, I don't have a login I can use.

Re: Fabric Data Pipeline - Copy activity stays in queued status

v-cboorla-msft — Mon, 22 Apr 2024 14:44:46 GMT

Hi Anonymous

No, I am asking you to provide the external source connection details. Please find the screenshots for reference.

This will help me in understand the ask and guide you in better way.

Thank you.

Re: Fabric Data Pipeline - Copy activity stays in queued status

Anonymous — Mon, 22 Apr 2024 14:57:57 GMT

v-cboorla-msft Here is the connection for the Source SQL Server.

Also see, the data copied from Source to staging area in the storage account.. This tells me it is able to connect to the source and copy the data to staging. It fails when it tries to connect to destination 'Warehouse'.

Re: Fabric Data Pipeline - Copy activity stays in queued status

v-cboorla-msft — Mon, 22 Apr 2024 15:07:11 GMT

Hi Anonymous

Try connecting to the souce by selecting Data gateway as "none" and run the pipeline.

If the issue still persists.

Please reach out to our support team to gain deeper insights and explore potential solutions. It's highly recommended that you reach out to our support team. Their expertise will be invaluable in suggesting the most appropriate approach.

Please go ahead and raise a support ticket to reach our support team:

https://support.fabric.microsoft.com/support

After creating a Support ticket please provide the ticket number as it would help us to track for more information.

Thank you.