Copy Data activity fails with KeyNotFoundException

[akaplan1-slcc]

I'm running a pipeline with several Copy Data activities.  All of the activities are copying from the same on-prem database to the same Fabric lakehouse schema (a different table for each activity).

One of these activities fails intermittently with the following error message:

ErrorCode=UserErrorWriteFailedFileOperation,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=The file operation is failed, upload file failed at path: '<workspace id>/<lakehouse id>/Tables/<schema name>/<table name>/<path to a specific file>.parquet'.,Source=Microsoft.DataTransfer.Common,''Type=System.Collections.Generic.KeyNotFoundException,Message=The given key was not present in the dictionary.,Source=mscorlib,'

It's always the same table that fails, but not always the same file.  The Copy Data activity overwrites the entire table (as do all the other activities in this pipeline).  There are two partition columns.  It's the largest table in this pipeline (over 30M rows, a bit under 30 MB when the activity finishes successfully), but not by much; at least two other tables are similar in size.

[v-tsaipranay]

Hi @vrsanaidu2025 ,

We haven’t received an update from you in some time. Could you please let us know if the issue has been resolved?
If you still require support, please let us know, we are happy to assist you.

Thank you.

[v-tsaipranay]

Hi @akaplan1-slcc ,

Sorry for the delay response. 

Yes, “Used parallel copies = 1” shows that the Copy activity was already running with minimal concurrency, so parallelism isn’t the issue.

Additionally, manually re-running the next day is different from enabling a retry on the activity. The retry policy works immediately within the same execution context, which can help resolve transient metadata issues like the KeyNotFoundException during partition overwrite. That’s why it’s recommended to set a small retry count, such as 2–3.

Since the file paths exist and no other pipeline writes to this table, the next thing to check is the stability of partition key values differences in casing, trailing spaces, special characters, or format changes can alter the physical folder name even if the values aren’t null.

Please try enabling the retry policy and check if the behaviour changes. Hope this helps. Please feel free to reach out for any further questions.

Thank you.

Thank you.

[akaplan1-slcc · Accepted solution]

That's helpful information; thank you.  The Copy activity ran fine with no errors this week, so it does indeed seem likely that the issue is a transient one.  I'll set the activity to 2 retries and see whether performance continues to be better for the next few eweks.

[v-tsaipranay]

Hi @akaplan1-slcc ,

Thank you for the update. Setting 2 retries is a suitable approach for this type of transient write behaviour. Please monitor the next few runs, and if it reoccurs, share the failed run’s Activity ID and timestamp so we can review it further.

Thank you.

[v-tsaipranay]

Hi @akaplan1-slcc ,

Thank you for confirming that there are no null values in your partition columns. Given the error details and what you've observed, it appears this is likely an intermittent metadata or file mapping issue occurring during the write phase of the Copy Data activity, rather than an issue with your source data.

A KeyNotFoundException can occur if the activity is unable to locate a required file or partition reference during overwrite operations. Possible causes include:

Concurrent or overlapping writes to the same Lakehouse table, overwrite cleanup and write processes running simultaneously for large tables, or a temporary service-side issue during metadata refresh or file handle resolution.

To help make the pipeline more stable, you can try these steps:

1. Confirm that no other activity or pipeline is writing to the same table or partition.
2. Lower the parallelism or split the copy process into smaller batches for that table.
3. Set up a retry policy in the Fault tolerance section of the Copy Data activity.
4. If a failure occurs, check whether the file path mentioned in the error still exists in the Lakehouse.

You can also review this helpful Microsoft documentation:

Options to get data into the Lakehouse - Microsoft Fabric | Microsoft Learn

Let me know what you find after checking the above points. Hope this helps. Please feel free to reach out for any further questions.

Thank you.

[akaplan1-slcc]

Notes on the four steps listed:

1. This is the only activity or pipeline that writes to this table.
2. "Degree of copy parallelism" is set to "Auto".  I checked the most recent failed pipeline runs, and both reported that "Used parallel copies" was 1 (see image below).  Does that mean that parallelism was already as low as possible?
3. I can set up a retry policy, but I hesitate to do that until I'm confident that the problem is likely to be fixed on a subsequent run.  The morning after the failure, I ran the pipeline again and got the same error; this suggests that re-running it immediately would not have helped either.
4. Yes, the files mentioned in the error message do exist in the lakehouse.

[vrsanaidu2025]

• If partition columns contain nulls or unexpected values, the dictionary used for partition mapping may fail.

• This could trigger a KeyNotFoundException during file path resolution.

[akaplan1-slcc]

I've just confirmed that the partition columns never contain null values.  I'm not sure what an unexpected value would look like; can you say more about that possibility?

If it matters, whenever I see this error message, the specific path is not a new or unexpected one - it's a file in a directory to which previous runs of the pipeline wrote other files with no problem.