Not able to write to datalake from virtual data gateway with Copy Data Task in Fabric

[kjetil_hamre]

Loading with dataflow2 pipelines works like a charm. However, when I try to do the same with copy-task in a datafactory pipeline (my preferred method) or a copy task - it just hangs for hours, doing nothing but exhausting my fabric capacity.

In the data factory pipeline, I am able preview the data in the source preview box. I am also able to write from one lakehouse table to another. I can also preview source in mapping.

[ex_kjetilh]

It is now hanging like this on a table with 17 rows..

[v-veshwara-msft]

Hi @kjetil_hamre ,

Thanks for the update and for sharing the screenshot.

Since the transfer phase completes and the delay occurs during the Lakehouse write, this could be related to schema alignment, permissions, or folder-level locking in the destination. You may want to test writing to a new table or folder to rule out metadata conflicts.

In some cases, updating the route table and ensuring WAN traffic is correctly routed through the firewall resolved the issue. If any custom routing or UDRs are in place, please check with your networking team to verify they’re not interfering.

Try simplifying the sink configuration by minimizing schema mapping and avoiding transformations during the copy. Test with a different small table to see if the issue is specific to that table or folder. Testing with the on-premises data gateway can also help determine if the issue is related to the Virtual Network Data Gateway.

Check for any error messages in the Monitoring hub by locating the pipeline run and reviewing the details of the Copy Data activity. This can reveal issues even if the status shows as “In Progress.”

If these checks don’t resolve the issue, please consider raising a support ticket with Microsoft so the engineering team can assist further.

Hope this helps. Please reach out for further assistance.

Thank you.

[ex_kjetilh]

The problem does not seem to have to do with sink - I have now switched to using an IntegrationRuntime on the source VM instead. Then it started working immediately. (but this is just a temporary solution as team decided to scrap integration runtime)

We are looking at the logs now.

[AntoineW]

Hello @kjetil_hamre,

Why does Copy Data task hang when writing to OneLake via Virtual Network Data Gateway?

• Root Cause:
  Copy Activity in Fabric pipelines can stall if:
  • The VNet Data Gateway is misconfigured or lacks outbound connectivity.
  • The Lakehouse sink settings or schema are invalid.
  • The scenario is not fully supported for Copy Activity (especially with large data or transformations).
  • Capacity is consumed but the write phase is blocked.

---

Key Points

1. Copy Activity relies on the gateway and Fabric connectors, so any network or schema issue can cause it to hang.
2. Capacity exhaustion happens because compute is allocated even if the write is stuck.

---

✅ How to fix it

• Check gateway configuration: Ensure VNet Data Gateway is healthy, linked to the right capacity, and has outbound HTTPS (443) access to Fabric endpoints.
• Validate Lakehouse sink: Confirm the destination table or folder exists and schema matches.
• Update pipeline settings:
  • Reduce parallelism or disable unnecessary staging.
  • If possible, enable Fast Copy (or switch to Dataflow Gen2 for heavy loads).
• Monitor capacity: Use the Fabric Capacity Metrics app to avoid CU waste.
• Permissions: Make sure the pipeline identity or service principal has Contributor access to the Lakehouse workspace.

Source : 

- https://learn.microsoft.com/en-us/fabric/data-factory/dataflows-gen2-fast-copy

- https://learn.microsoft.com/en-us/fabric/data-factory/copy-job-with-virtual-network-data-gateway

- https://learn.microsoft.com/en-us/data-integration/vnet/use-data-gateways-pipeline-fabric

- https://learn.microsoft.com/en-us/fabric/data-factory/gateway-considerations-output-destinations

- https://learn.microsoft.com/en-us/fabric/fundamentals/decision-guide-pipeline-dataflow-spark

Hope it can help you ! 

Best regards,

Antoine

[kjetil_hamre]

Thank you for very prompt answer. I think it must be the VNET. Investigating.

[v-veshwara-msft]

Hi @kjetil_hamre ,
Thanks for reaching out to Microsoft Fabric Community and thanks for the update.

Good to know you are investigating the VNET side.

As @AntoineW explained, Copy Activity in Fabric pipelines depends on the Virtual Network Data Gateway for outbound connectivity to Fabric endpoints. If the gateway is not correctly configured or restricted on outbound HTTPS (443), the copy task may hang while still consuming capacity. It is also important to make sure the Lakehouse sink is valid, with schema alignment and permissions in place, otherwise the write phase can remain blocked even though preview works.

There have been similar reports where writes to Lakehouse failed due to routing. One case was resolved by adding a microsoft.storage service endpoint to the delegated gateway subnet, details here: Solved: Re: VNET data gateway cannot write to lakehouse - Microsoft Fabric Community 

Microsoft also announced GA support for Virtual Network Data Gateway with Fabric Pipeline, Dataflow Gen2 Fast Copy, and Copy Job, which may improve reliability in such cases.

Virtual Network Data Gateway Support for Fabric Pipeline, Dataflow Gen2 Fast Copy, and Copy Job (Gen...

As an alternative, you could also test using the on-premises data gateway, which is supported in Fabric pipelines and Dataflow Gen2. 

How to access on-premises data sources in Data Factory - Microsoft Fabric | Microsoft Learn

Hope this helps. Please reach out for further assistance.

Thank you.

[ex_kjetilh]

Thank you. We have found some issues with the virtual gateway and DNS, and got it to work for a brief period, but now not working again. Still investigating.