topic Not able to write to datalake from virtual data gateway with Copy Data Task in Fabric in Pipelines

Not able to write to datalake from virtual data gateway with Copy Data Task in Fabric

kjetil_hamre — Thu, 25 Sep 2025 11:00:48 GMT

Loading with dataflow2 pipelines works like a charm. However, when I try to do the same with copy-task in a datafactory pipeline (my preferred method) or a copy task - it just hangs for hours, doing nothing but exhausting my fabric capacity.

In the data factory pipeline, I am able preview the data in the source preview box. I am also able to write from one lakehouse table to another. I can also preview source in mapping.

Re: Not able to write to datalake from virtual data gateway with Copy Data Task in Fabric

AntoineW — Thu, 25 Sep 2025 11:32:34 GMT

Hello kjetil_hamre,

Why does Copy Data task hang when writing to OneLake via Virtual Network Data Gateway?

Root Cause:
Copy Activity in Fabric pipelines can stall if:
- The VNet Data Gateway is misconfigured or lacks outbound connectivity.
- The Lakehouse sink settings or schema are invalid.
- The scenario is not fully supported for Copy Activity (especially with large data or transformations).
- Capacity is consumed but the write phase is blocked.

Key Points

Copy Activity relies on the gateway and Fabric connectors, so any network or schema issue can cause it to hang.
Capacity exhaustion happens because compute is allocated even if the write is stuck.

✅ How to fix it

Check gateway configuration: Ensure VNet Data Gateway is healthy, linked to the right capacity, and has outbound HTTPS (443) access to Fabric endpoints.
Validate Lakehouse sink: Confirm the destination table or folder exists and schema matches.
Update pipeline settings:
- Reduce parallelism or disable unnecessary staging.
- If possible, enable Fast Copy (or switch to Dataflow Gen2 for heavy loads).
Monitor capacity: Use the Fabric Capacity Metrics app to avoid CU waste.
Permissions: Make sure the pipeline identity or service principal has Contributor access to the Lakehouse workspace.

Source :

- https://learn.microsoft.com/en-us/fabric/data-factory/dataflows-gen2-fast-copy

- https://learn.microsoft.com/en-us/fabric/data-factory/copy-job-with-virtual-network-data-gateway

- https://learn.microsoft.com/en-us/data-integration/vnet/use-data-gateways-pipeline-fabric

- https://learn.microsoft.com/en-us/fabric/data-factory/gateway-considerations-output-destinations

- https://learn.microsoft.com/en-us/fabric/fundamentals/decision-guide-pipeline-dataflow-spark

Hope it can help you !

Best regards,

Antoine

Re: Not able to write to datalake from virtual data gateway with Copy Data Task in Fabric

kjetil_hamre — Thu, 25 Sep 2025 12:39:49 GMT

Thank you for very prompt answer. I think it must be the VNET. Investigating.

Re: Not able to write to datalake from virtual data gateway with Copy Data Task in Fabric

v-veshwara-msft — Tue, 30 Sep 2025 05:38:26 GMT

Hi kjetil_hamre ,
Thanks for reaching out to Microsoft Fabric Community and thanks for the update.

Good to know you are investigating the VNET side.

As AntoineW explained, Copy Activity in Fabric pipelines depends on the Virtual Network Data Gateway for outbound connectivity to Fabric endpoints. If the gateway is not correctly configured or restricted on outbound HTTPS (443), the copy task may hang while still consuming capacity. It is also important to make sure the Lakehouse sink is valid, with schema alignment and permissions in place, otherwise the write phase can remain blocked even though preview works.

There have been similar reports where writes to Lakehouse failed due to routing. One case was resolved by adding a microsoft.storage service endpoint to the delegated gateway subnet, details here: Solved: Re: VNET data gateway cannot write to lakehouse - Microsoft Fabric Community

Microsoft also announced GA support for Virtual Network Data Gateway with Fabric Pipeline, Dataflow Gen2 Fast Copy, and Copy Job, which may improve reliability in such cases.

Virtual Network Data Gateway Support for Fabric Pipeline, Dataflow Gen2 Fast Copy, and Copy Job (Generally Available) | Microsoft Fabric Blog | Microsoft Fabric

As an alternative, you could also test using the on-premises data gateway, which is supported in Fabric pipelines and Dataflow Gen2.

How to access on-premises data sources in Data Factory - Microsoft Fabric | Microsoft Learn

Hope this helps. Please reach out for further assistance.

Thank you.

Re: Not able to write to datalake from virtual data gateway with Copy Data Task in Fabric

ex_kjetilh — Tue, 30 Sep 2025 06:59:30 GMT

Thank you. We have found some issues with the virtual gateway and DNS, and got it to work for a brief period, but now not working again. Still investigating.

Re: Not able to write to datalake from virtual data gateway with Copy Data Task in Fabric

ex_kjetilh — Tue, 30 Sep 2025 07:38:40 GMT

It is now hanging like this on a table with 17 rows..

Re: Not able to write to datalake from virtual data gateway with Copy Data Task in Fabric

v-veshwara-msft — Wed, 01 Oct 2025 09:13:30 GMT

Hi kjetil_hamre ,

Thanks for the update and for sharing the screenshot.

Since the transfer phase completes and the delay occurs during the Lakehouse write, this could be related to schema alignment, permissions, or folder-level locking in the destination. You may want to test writing to a new table or folder to rule out metadata conflicts.

In some cases, updating the route table and ensuring WAN traffic is correctly routed through the firewall resolved the issue. If any custom routing or UDRs are in place, please check with your networking team to verify they’re not interfering.

Try simplifying the sink configuration by minimizing schema mapping and avoiding transformations during the copy. Test with a different small table to see if the issue is specific to that table or folder. Testing with the on-premises data gateway can also help determine if the issue is related to the Virtual Network Data Gateway.

Check for any error messages in the Monitoring hub by locating the pipeline run and reviewing the details of the Copy Data activity. This can reveal issues even if the status shows as “In Progress.”

If these checks don’t resolve the issue, please consider raising a support ticket with Microsoft so the engineering team can assist further.

Hope this helps. Please reach out for further assistance.

Thank you.

Re: Not able to write to datalake from virtual data gateway with Copy Data Task in Fabric

ex_kjetilh — Wed, 01 Oct 2025 09:34:50 GMT

The problem does not seem to have to do with sink - I have now switched to using an IntegrationRuntime on the source VM instead. Then it started working immediately. (but this is just a temporary solution as team decided to scrap integration runtime)

We are looking at the logs now.

Re: Not able to write to datalake from virtual data gateway with Copy Data Task in Fabric

v-veshwara-msft — Mon, 06 Oct 2025 06:50:56 GMT

Hi ex_kjetilh ,
Thanks for the update. Since switching to the Integration Runtime on the source VM resolved the issue for now, it looks like the problem is tied to the Virtual Network Data Gateway or its network setup.

Have you found anything new while going through the logs or testing further? If you need assistance reviewing your findings or planning further steps, please let us know.

Thank you.

Re: Not able to write to datalake from virtual data gateway with Copy Data Task in Fabric

v-veshwara-msft — Fri, 10 Oct 2025 05:51:11 GMT

Hi ex_kjetilh ,

Just checking in to see if you were able to gather anything from the logs or further testing after switching to the Integration Runtime. If you need any further assistance, feel free to reach out.

Thank you.

Re: Not able to write to datalake from virtual data gateway with Copy Data Task in Fabric

v-veshwara-msft — Tue, 14 Oct 2025 05:19:54 GMT

Hi kjetil_hamre ,
We wanted to kindly follow up regarding your query. If you need any further assistance, please reach out.
Thank you.