Forum Discussion
Copy Job Triggered Twice When Executed from Pipeline (Foreach)
Hi D4zk,
Thanks for the detailed explanation.From a Copy Job perspective, the execution details that are exposed are currently limited to the activity-level logs (as you’ve already observed). These logs typically show the overall success/failure status and high-level error messages, but they do not always provide full visibility into intermediate states such as LSN evaluation or CDC window handling.
Because of this, scenarios like:
- One execution succeeding
- A subsequent execution failing
may not always have enough diagnostic detail available directly within the Copy Job monitoring experience.
You might consider validating the source query independently (in SQL Server) to better understand how it behaves across executions. If deeper visibility or control is required, implementing a custom pipeline can help with explicit state tracking and logging.
I hope this information helps. Please do let us know if you have any further queries.
Thank you!
Hi v-abhinavmu
I understand the root cause of the issue itself. What I would like to clarify is why the job was executed twice — one run succeeded and the second run failed.
From my analysis, the failure happened because there were no new CDC changes, but the logic still attempted to proceed since from_lsn (derived value) became greater than fn_cdc_get_max_lsn(). This resulted in the error shown. However, this still raises concerns for me.
My main worry is whether the Copy Job is correctly tracking and picking up the last processed LSN. The double execution (success followed by failure) makes me unsure about the reliability and determinism of the Copy Job’s CDC state handling.
Because of these concerns—and also because the monitoring and execution history of Copy Job does not provide sufficient or meaningful diagnostic details—I am considering building and using my own custom pipeline for CDC tables instead of the built‑in Copy Job. At the moment, there seem to be too many edge cases, bugs, or opaque behaviors - I still had another opening issue post - that are difficult to validate or troubleshoot.
I’d appreciate your thoughts or confirmation on this behavior.
Community SupportHi D4zk,
Thanks for the detailed explanation.
From a Copy Job perspective, the execution details that are exposed are currently limited to the activity-level logs (as you’ve already observed). These logs typically show the overall success/failure status and high-level error messages, but they do not always provide full visibility into intermediate states such as LSN evaluation or CDC window handling.
Because of this, scenarios like:
- One execution succeeding
- A subsequent execution failing
may not always have enough diagnostic detail available directly within the Copy Job monitoring experience.
You might consider validating the source query independently (in SQL Server) to better understand how it behaves across executions. If deeper visibility or control is required, implementing a custom pipeline can help with explicit state tracking and logging.
I hope this information helps. Please do let us know if you have any further queries.
Thank you!