QUESTION::PIPELINE::COPY DATA::INTEGRITY

[Element115]

I am curious to learn how do any of you guys who are using pipelines and a Copy activity check that the data is actually copied correctly, that is, no missing rows, or offset rows in datetime, and so on?

I ask because I noticed something odd today, but before I start shouting 'BUG!', and because it is going to take me a while to double check everything (well, one thing already checked, I do nothing in the pipeline to transform the incoming data, so if it turns out that there is an issue with the pipeline, then it is with Copy data or the data gateway (using an on-prem source) or some weird network glitch).

[Element115]

I got this support ticket currently opened:  2406040040000671.

All, beware that ID columns get out of sequence and new numbers seem to be printed by either the 'Copy data' activity or the lakehouse process or...?  In any case, new ID numbers appear in the sequence (no gaps) that do not exist at source!  So you'd be advised to check your data integrity because if the backend process does whatever it wants with your incoming data, then you have no data integrity guarantees.  Go figure that one.

[v-cboorla-msft]

Hi @Element115 

Thanks for using Microsoft Fabric Community.

When you move data from source to destination store, the copy activity provides an option for you to do further data consistency verification to ensure the data is not only successfully copied from source to destination store, but also verified to be consistent between source and destination store. Once inconsistent files have been found during the data movement, you can either abort the copy activity or continue to copy the rest by enabling fault tolerance setting to skip inconsistent files.

Please refer the documentation for more details : Data consistency verification in copy activity.

I hope this information helps. Please do let us know if you have any further queries.

Thank you.

[Element115]

That's all nice and dandy but where do you set this option in the GUI?  There is nothing I can see.