Forum Discussion
Copy Data activity - ErrorCode=FailedToUpsertDataIntoDeltaTable
Hi everyone!
I have a pipeline with a Copy Data activity that is loading data into a Delta table in a Lakehouse. The first ingestion runs successfully, but when I run the pipeline for a second time, the copy activity fails during the upsert operation with the following error:
ErrorCode=FailedToUpsertDataIntoDeltaTable,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=Hit an error when upsert data to table in Lakehouse. Error message: Could not load file or assembly 'Microsoft.Bcl.HashCode, Version=1.0.0.0, Culture=neutral, PublicKeyToken=xxxxx' or one of its dependencies. El sistema no puede encontrar el archivo especificado.,Source=Microsoft.DataTransfer.Connectors.LakehouseTableConnector,''Type=System.IO.FileNotFoundException,Message=Could not load file or assembly 'Microsoft.Bcl.HashCode, Version=1.0.0.0, Culture=neutral, PublicKeyToken=xxxxx' or one of its dependencies. El sistema no puede encontrar el archivo especificado.,Source=TypeExtension,'
Has anyone encountered the same issue or have any ideas on what might be causing this? Any insights or troubleshooting steps would be greatly appreciated.
Thanks in advance!
Hi KerenLopez ,
Thank you for reaching out to the Microsoft Community Forum.
Please try below things to fix the issue.
1. Turn on “Enable staging” for the Copy activity:
Open your pipeline --> Copy data activity --> Settings tab. Turn on Enable staging. Re-run the pipeline.
2. Use a staging table and MERGE in a Notebook:
Change sink to Files --> Parquet or a staging Delta table. Add a Notebook activity after the copy with a simple Delta MERGE INTO from staging to the target table.
Please check below settings.1. Make sure the sink is a Lakehouse Table (Delta) rather than the SQL endpoint of the Lakehouse. The SQL endpoint in Fabric is read-only and can’t be used as a Copy sink for upsert.
2. Verify the key columns under Table action --> Upsert are correctly mapped and present in the incoming dataset. they will cause other upsert failures.
3. Avoid temporary tables in source-side stored procedures when possible, historical issues show these can behave differently across connectors and modes. If you need, test with a simplified SELECT to isolate whether the failure is due to source-side temp objects or the sink upsert path.
Please refer below links.
Solved: Copy activity fails to save data as delta table bu... - Microsoft Fabric Community
Solved: Failed to Upsert Data into Lakehouse Delta Table –... - Microsoft Fabric Community
I hope this information helps. Please do let us know if you have any further queries.
Regards,
Dinesh
3 Replies
Hi KerenLopez ,
Thank you for reaching out to the Microsoft Community Forum.
Please try below things to fix the issue.
1. Turn on “Enable staging” for the Copy activity:
Open your pipeline --> Copy data activity --> Settings tab. Turn on Enable staging. Re-run the pipeline.
2. Use a staging table and MERGE in a Notebook:
Change sink to Files --> Parquet or a staging Delta table. Add a Notebook activity after the copy with a simple Delta MERGE INTO from staging to the target table.
Please check below settings.1. Make sure the sink is a Lakehouse Table (Delta) rather than the SQL endpoint of the Lakehouse. The SQL endpoint in Fabric is read-only and can’t be used as a Copy sink for upsert.
2. Verify the key columns under Table action --> Upsert are correctly mapped and present in the incoming dataset. they will cause other upsert failures.
3. Avoid temporary tables in source-side stored procedures when possible, historical issues show these can behave differently across connectors and modes. If you need, test with a simplified SELECT to isolate whether the failure is due to source-side temp objects or the sink upsert path.
Please refer below links.
Solved: Copy activity fails to save data as delta table bu... - Microsoft Fabric Community
Solved: Failed to Upsert Data into Lakehouse Delta Table –... - Microsoft Fabric Community
I hope this information helps. Please do let us know if you have any further queries.
Regards,
Dinesh
Hi KerenLopez ,
We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. And, if you have any further query do let us know.
Regards,
Dinesh
Hi v-dineshya,
Apologies for the delayed response 😅.
I wanted to let you know that the first option you suggested — “Enable staging” in the Copy activity — worked perfectly and resolved the issue.
Thank you very much for your support and for the clear explanation. I really appreciate your help.
Regards,
Keren.
