Forum Discussion
Anonymous
1 year agoNot applicable
Copy Data Nulling out Destination Delta Tables On Source Connection Error
In a data pipeline, if you use a copy data activity and use the 'overwrite' table action, when that activity runs, if it fails to connect to the source, it will overwrite the destination Lakehouse De...
- Anonymous1 year ago
Hi Anonymous
Before running the copy data activity, add a pre-validation step to check the availability of the source. If the source is unavailable, you can stop the pipeline execution.
For example, the process could be:
- Add a Get Metadata activity to connect to the data source and check some meta data from it. You can also try Lookup activity or other activities according to the data source. For example, if a table is to be copied, get the number of rows of the source table.
- Add an If Condition activity to check the output of previous activity. If the output is desired indicating the data source is ready, returns True. Otherwise returns False. For exmaple, check whether the number of table rows is greater than 0.
- In previous step's If Condition activity, execute the Copy data activity in True scenario.
Hope this would be helpful.
Best Regards,
Jing
If this post helps, please Accept it as Solution to help other members find it. Appreciate your Kudos!
FabianSchut
Solution Sage
1 year agoI did not check this, but this may be resolved if you enable staging in the copy activity.