Forum Discussion
Copy Data activity fails with KeyNotFoundException
That's helpful information; thank you. The Copy activity ran fine with no errors this week, so it does indeed seem likely that the issue is a transient one. I'll set the activity to 2 retries and see whether performance continues to be better for the next few eweks.
Hi akaplan1-slcc ,
Thank you for confirming that there are no null values in your partition columns. Given the error details and what you've observed, it appears this is likely an intermittent metadata or file mapping issue occurring during the write phase of the Copy Data activity, rather than an issue with your source data.
A KeyNotFoundException can occur if the activity is unable to locate a required file or partition reference during overwrite operations. Possible causes include:
Concurrent or overlapping writes to the same Lakehouse table, overwrite cleanup and write processes running simultaneously for large tables, or a temporary service-side issue during metadata refresh or file handle resolution.
To help make the pipeline more stable, you can try these steps:
- Confirm that no other activity or pipeline is writing to the same table or partition.
- Lower the parallelism or split the copy process into smaller batches for that table.
- Set up a retry policy in the Fault tolerance section of the Copy Data activity.
- If a failure occurs, check whether the file path mentioned in the error still exists in the Lakehouse.
You can also review this helpful Microsoft documentation:
Options to get data into the Lakehouse - Microsoft Fabric | Microsoft Learn
Let me know what you find after checking the above points. Hope this helps. Please feel free to reach out for any further questions.
Thank you.
Notes on the four steps listed:
- This is the only activity or pipeline that writes to this table.
- "Degree of copy parallelism" is set to "Auto". I checked the most recent failed pipeline runs, and both reported that "Used parallel copies" was 1 (see image below). Does that mean that parallelism was already as low as possible?
- I can set up a retry policy, but I hesitate to do that until I'm confident that the problem is likely to be fixed on a subsequent run. The morning after the failure, I ran the pipeline again and got the same error; this suggests that re-running it immediately would not have helped either.
- Yes, the files mentioned in the error message do exist in the lakehouse.