Forum Discussion
Fabric Copy Job Timeout
Thanks, I agree with your first statement but its a business requirement that needs to get done in a very short timeframe, and fabric's copy job/cdc feature proved to be the quickest and most efficient option.
I tried using a Fabric pipeline to seed the initial data onto sql2025 then enable cdc to only capture the deltas, however CDC just tries to copy the full dataset and runs into that same timeout error.
I've resolved to using an incremental pipeline instead using a watermark column to migrate new rows.
Advocate IIIHi joel_delima,
this looks like a gateway-side timeout, not an issue with your Copy Job logic itself. A few things point that way:
- The error source is Microsoft.DataTransfer.DataContracts, which is the data movement/gateway layer, and "Dispose resource timeout" typically means the connection/resource couldn't be cleanly closed within the allotted time, common when a large initial load keeps a connection open far longer than smaller incremental runs.
- It only fails on the first run on large tables (100M+ rows) because incremental runs after that only move the delta (via CDC), much smaller payloads, so the gateway connection doesn't stay open nearly as long.
Likely causes and fixes:
- On-prem Data Gateway timeout settings - the gateway has its own internal query timeout separate from the Copy Job's settings. Check Microsoft.PowerBI.DataMovement.Pipeline.GatewayCore.dll.config on the gateway machine for timeout values (similar to the MashupDefaultPoolContainerMaxWorkingSetInMB type settings), these often need to be raised for very large initial loads.
- Gateway resource exhaustion - 100M+ row initial loads can exhaust memory/connections on the gateway VM over a 40-minute window. Check gateway machine resource usage (CPU/RAM) during the run.
- Split the initial load - rather than relying on the Copy Job to pull the full 100M+ rows in one shot, do a one-time manual bulk load (e.g., via a Pipeline Copy activity with partitioning, or BCP/bulk insert) to seed the table, then let the Fabric Copy Job take over for incremental CDC sync going forward. This sidesteps the timeout entirely for the problematic first run.
Option 3 is generally the most reliable fix reported for "first full load fails, incremental works fine" patterns with CDC-based pipelines.
Thank you, I will review those potential fixes.