Forum Discussion
Copy Job Triggered Twice When Executed from Pipeline (Foreach)
However, it is executed twice. One execution completes successfully, but the other fails with the error message below: Failure happened on 'Source' side. ErrorCode=FailedToUpsertDataIntoDeltaTable,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=Hit an error when upsert data to table in Lakehouse. Error message: An insufficient number of arguments were supplied for the procedure or function cdc.fn_cdc_get_net_changes_ ...
Hi D4zk,
Thanks for the detailed explanation.From a Copy Job perspective, the execution details that are exposed are currently limited to the activity-level logs (as you’ve already observed). These logs typically show the overall success/failure status and high-level error messages, but they do not always provide full visibility into intermediate states such as LSN evaluation or CDC window handling.
Because of this, scenarios like:
- One execution succeeding
- A subsequent execution failing
may not always have enough diagnostic detail available directly within the Copy Job monitoring experience.
You might consider validating the source query independently (in SQL Server) to better understand how it behaves across executions. If deeper visibility or control is required, implementing a custom pipeline can help with explicit state tracking and logging.
I hope this information helps. Please do let us know if you have any further queries.
Thank you!
8 Replies
Hi @nielsvdc I saw your comment on topic Solved: Copy Jobs Failing Because Multiple are Running - Microsoft Fabric Community Could you help to give me more information about that?
v-abhinavmuCommunity Support
Hi D4zk,
Thanks for reaching out to the Microsoft Fabric Community forum.I reproduced a similar scenario and observed that when the ForEach activity runs in parallel, multiple executions start simultaneously, which can lead to overlapping runs. After switching ForEach to sequential execution (or setting batch count = 1), the activities ran one after another and the issue was resolved.
As a best practice, for dynamic or metadata-driven scenarios, consider using a Copy Data activity, as they are better suited for parameterization and pipeline orchestration.
I hope this information helps. Please do let us know if you have any further queries.
Thank youThank you for your response. But after switching to the sequence, I am still encountering this issue:
Once copy job has been executed 2 times:
And the error message you can see in the image is: Failure happened on 'Source' side. ErrorCode=FailedToUpsertDataIntoDeltaTable,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=Hit an error when upsert data to table in Lakehouse. Error message: An insufficient number of arguments were supplied for the procedure or function cdc.fn_cdc_get_net_changes_ ... .,Source=Microsoft.DataTransfer.Connectors.LakehouseTableConnector,''Type=System.Data.SqlClient.SqlException,Message=An insufficient number of arguments were supplied for the procedure or function cdc.fn_cdc_get_net_changes_ ... .,Source=.Net SqlClient Data Provider,SqlErrorNumber=313,Class=16,ErrorCode=-2146232060,State=3,Errors=[{Class=16,Number=313,State=3,Message=An insufficient number of arguments were supplied for the procedure or function cdc.fn_cdc_get_net_changes_ ... .,},],'
- v-abhinavmu
Hi D4zk,
Thanks for the update.From the error in your previous message:
“An insufficient number of arguments were supplied for the procedure cdc.fn_cdc_get_net_changes…”,
this usually indicates that the CDC function is not receiving the required parameters.This function expects values like start LSN and end LSN, and if they are missing, null, or not passed correctly, it can result in this error.
You might try checking the following:
1. How the CDC function is being called in your source query or Copy Job
2. Whether the required parameters (LSN values) are being passed correctly
3. If these values are dynamic, ensure they are not empty at runtime
4. Try running the same query directly in SQL Server to validate it
If the issue still persists, as a quick troubleshooting step, you could try recreating the setup from scratch.
Since one run succeeds and another fails, it may also help to check if different parameter values are being passed in each execution.
For more details, please refer to the official documentation:
cdc.fn_cdc_get_net_changes_<capture_instance> (Transact-SQL) - SQL Server | Microsoft Learn
Please give it a try and let me know how it goes.
Thank you!
- v-abhinavmu
Hi D4zk,
I wanted to check if you had the opportunity to review the information provided. Please feel free to contact us if you have any further questions.
Thank you. - v-abhinavmu
Hi D4zk,
As we haven’t heard back from you, we wanted to kindly follow up to check if the suggestions provided by the community members for the issue worked. Please feel free to contact us if you have any further questions.
Thank you!