Forum Discussion

Element115's avatar
Element115
Memorable Member
2 years ago

QUESTION::PIPELINE::COPY DATA::INTEGRITY

I am curious to learn how do any of you guys who are using pipelines and a Copy activity check that the data is actually copied correctly, that is, no missing rows, or offset rows in datetime, and so...
data engineering
v-cboorla-msft's avatar
v-cboorla-msft
Microsoft Employee
2 years ago

Hi Element115 

 

Thanks for using Microsoft Fabric Community.

When you move data from source to destination store, the copy activity provides an option for you to do further data consistency verification to ensure the data is not only successfully copied from source to destination store, but also verified to be consistent between source and destination store. Once inconsistent files have been found during the data movement, you can either abort the copy activity or continue to copy the rest by enabling fault tolerance setting to skip inconsistent files.

Please refer the documentation for more details : Data consistency verification in copy activity.

 

I hope this information helps. Please do let us know if you have any further queries.

 

Thank you.

Element115's avatar
Element115
Memorable Member
2 years ago

That's all nice and dandy but where do you set this option in the GUI?  There is nothing I can see.