topic Re: Copy activity in Microsoft Fabric pipeline not enforcing schema and not throwing error in Pipelines

Copy activity in Microsoft Fabric pipeline not enforcing schema and not throwing error

Fabamik — Fri, 04 Apr 2025 10:41:00 GMT

We are using copy data activity in Pipelines in Microsoft Fabric. Copyind data using Dataverse connector as source and lakehouse delta table as connector. If source and destination schema is not matching copy activity is not failing sctivity while writing in destination delta table. But if you try to read that table then it will throw error of schema mismatch. So how to ensure that copy data activity will fail if schema mismatch between source and destination. Source column is string and destination is int

Re: Copy activity in Microsoft Fabric pipeline not enforcing schema and not throwing error

nilendraFabric — Fri, 04 Apr 2025 14:33:16 GMT

Hi Fabamik

schema drift is not available in Microsoft Fabric, which means the Copy activity doesn't automatically validate or enforce schema compatibility between source and destination.

you can enable fault tolerance in the Copy activity to handle schema mismatches and other data incompatibilities by :

Re: Copy activity in Microsoft Fabric pipeline not enforcing schema and not throwing error

Anonymous — Mon, 07 Apr 2025 12:49:47 GMT

Hi Fabamik,

Thanks nilendraFabric for Addressing the issue.

we would like to follow up to see if the solution provided by the super user resolved your issue. Please let us know if you need any further assistance.
If our super user response resolved your issue, please mark it as "Accept as solution" and click "Yes" if you found it helpful.

Regards,
Vinay Pabbu

Re: Copy activity in Microsoft Fabric pipeline not enforcing schema and not throwing error

Fabamik — Tue, 08 Apr 2025 07:27:38 GMT

Thanks nilendraFabric for your reply. Yes, as you mentioned schema drift is not available in Microsoft Fabric but what I observed I tried with 3 different connectors/sources in copy acitivity and following is the output -

1. In copy activity using Dataverse as source- here copy activity not throwing any error and completing copy activity successfully and when we are trying to read that delta table it is throwing error of schema mismatch

2. In copy activity using Lakehouse table as source - here copy activity throwing error during coping only and failing that copy activity

3. In copy activity using Lakehouse delimted file as source - here copy activity throwing error during coping only and failing that copy activity

It seems like copy activity showing different behaviour w.r.to different sources

Re: Copy activity in Microsoft Fabric pipeline not enforcing schema and not throwing error

Anonymous — Fri, 11 Apr 2025 11:51:04 GMT

Hi Fabamik,

Microsoft Fabric’s Copy activity exhibits different behaviors based on the source type due to how schema handling and data serialization work under the hood.
https://learn.microsoft.com/en-us/azure/data-factory/copy-activity-schema-and-type-mapping

Regards,

Vinay Pabbu

Re: Copy activity in Microsoft Fabric pipeline not enforcing schema and not throwing error

Anonymous — Mon, 14 Apr 2025 13:34:04 GMT

Hi @Fabamik,

Regards,
Vinay Pabbu

Re: Copy activity in Microsoft Fabric pipeline not enforcing schema and not throwing error

Anonymous — Thu, 17 Apr 2025 12:06:58 GMT

Hi @Fabamik,

Regards,
Vinay Pabbu

Re: Copy activity in Microsoft Fabric pipeline not enforcing schema and not throwing error

Fabamik — Thu, 17 Apr 2025 15:33:00 GMT

Thanks Anonymous for your resposne. It seems like for every source we need to check thoroughly and then decide laod strategy. We can't assume anything during design

Re: Copy activity in Microsoft Fabric pipeline not enforcing schema and not throwing error

Anonymous — Mon, 21 Apr 2025 12:24:30 GMT

Hi Fabamik,

Absolutely, each source needs a detailed review to define the right load strategy. We can't assume anything upfront, as each one may have unique schema requirements.

Regards,

Vinay Pabbu