Copy activity in Microsoft Fabric pipeline not enforcing schema and not throwing error

[Fabamik]

We are using copy data activity in Pipelines in Microsoft Fabric. Copyind data using Dataverse connector as source and lakehouse delta table as connector. If source and destination schema is not matching copy activity is not failing sctivity while writing in destination delta table. But if you try to read that table then it will throw error of schema mismatch. So how to ensure that copy data activity will fail if schema mismatch between source and destination. Source column is string and destination is int

[nilendraFabric · Accepted solution]

Hi @Fabamik 

schema drift is not available in Microsoft Fabric, which means the Copy activity doesn't automatically validate or enforce schema compatibility between source and destination.

you can enable fault tolerance in the Copy activity to handle schema mismatches and other data incompatibilities by :

[Fabamik]

Thanks @v-vpabbu  for your resposne. It seems like for every source we need to check thoroughly and then decide laod strategy. We can't assume anything during design

[v-vpabbu]

Hi @Fabamik,

Absolutely, each source needs a detailed review to define the right load strategy. We can't assume anything upfront, as each one may have unique schema requirements.

Regards,

Vinay Pabbu

[v-vpabbu]

Hi @Fabamik,

Thanks @nilendraFabric for Addressing the issue.

we would like to follow up to see if the solution provided by the super user resolved your issue. Please let us know if you need any further assistance.
If our super user response resolved your issue, please mark it as "Accept as solution" and click "Yes" if you found it helpful.

Regards,
Vinay Pabbu

[v-vpabbu]

Hi @Fabamik,

we would like to follow up to see if the solution provided by the super user resolved your issue. Please let us know if you need any further assistance.
If our super user response resolved your issue, please mark it as "Accept as solution" and click "Yes" if you found it helpful.

Regards,
Vinay Pabbu

[v-vpabbu]

Hi @Fabamik,

we would like to follow up to see if the solution provided by the super user resolved your issue. Please let us know if you need any further assistance.
If our super user response resolved your issue, please mark it as "Accept as solution" and click "Yes" if you found it helpful.

Regards,
Vinay Pabbu

[nilendraFabric · Accepted solution]

Hi @Fabamik 

schema drift is not available in Microsoft Fabric, which means the Copy activity doesn't automatically validate or enforce schema compatibility between source and destination.

you can enable fault tolerance in the Copy activity to handle schema mismatches and other data incompatibilities by :

[Fabamik]

Thanks @nilendraFabric for your reply. Yes, as you mentioned schema drift is not available in Microsoft Fabric but what I observed I tried with 3 different connectors/sources in copy acitivity and following is the output -

1. In copy activity using Dataverse as source- here copy activity not throwing any error and completing copy activity successfully and when we are trying to read that delta table it is throwing error of schema mismatch

2. In copy activity using Lakehouse table as source  - here copy activity throwing error during coping only and failing that copy activity

3. In copy activity using Lakehouse delimted file as source  - here copy activity throwing error during coping only and failing that copy activity

It seems like copy activity showing different behaviour w.r.to different sources

[v-vpabbu · Accepted solution]

Hi @Fabamik,

Microsoft Fabric’s Copy activity exhibits different behaviors based on the source type due to how schema handling and data serialization work under the hood.
https://learn.microsoft.com/en-us/azure/data-factory/copy-activity-schema-and-type-mapping 

Regards,

Vinay Pabbu