topic Re: How to handle schema drift? in Pipelines

How to handle schema drift?

dhorrall — Sun, 27 Aug 2023 12:54:43 GMT

In legacy datafactory there were options to explicitly allow schema drift. I do not see that in Fabric. Am I missing it?

For example,

I was just doing some random testing of loading historical blob text files into a 'table' in Fabric to kick the tires
I had created some intermediary parquet files from the text files to get practice with that
I then attempted to load that parquet files to a 'table', and got the error..'Source column is not defined in delta meta data'
Obviously because files have columns that evolved over time

I don't see a straight-forward way to handle this.

Re: How to handle schema drift?

ajarora — Tue, 29 Aug 2023 01:29:42 GMT

How do you expect the schema variation to have taken effect ?

There can be many situations possible, column added, column dropped, or column type changed.

Re: How to handle schema drift?

dhorrall — Sat, 02 Sep 2023 13:30:24 GMT

Probably all of the above. Current 'datafactory' has checkbox to handle. I see nothing like this in Fabric. This was the basis of my question.

Re: How to handle schema drift?

ajarora — Tue, 05 Sep 2023 23:16:37 GMT

What you are referring to is possible through Azure Data Factory Mapping Dataflows, but they are not available in Fabric. Perhaps you want to try out Fabric Dataflows and see if it applies as it is ?

In terms of what copy activity allows, if your destination table already exists, and the data you are writing has a column missing, it will be defaulted to null (default value) when writing to destination. If there is a new column, or if a column is not typecastable to the destination type, then this is treated as a bad row, and you can either skip writing this bad row (and log it into a temporary storage to be processed later), or fail the operation (the default).

Re: How to handle schema drift?

GraceGu — Wed, 06 Sep 2023 02:03:03 GMT

Suppose the ask is for explict schema mapping in copy activity exists in ADF today. Editing mapping for Lakehouse destination will be coming in 1-2 month. @dhorrall what destination you are looking for?

Re: How to handle schema drift?

Anonymous — Mon, 18 Sep 2023 15:18:48 GMT

Is the ADF Mapping Dataflow coming to Fabric ?
We have the same kind of requirement with json files as source, evolving with new attributes, we need to have the schema drift available

Re: How to handle schema drift?

NeedAUserName — Wed, 27 Sep 2023 18:22:35 GMT

Any answer on this? We have the same requirements and need pipelines to be able to handle schema drift as it is under ADF.

Re: How to handle schema drift?

Jreed_7474 — Wed, 27 Mar 2024 19:20:52 GMT

Any update on if schema drift will be added in the futre? especially being it looks like that functionality exsists in ADF?

Re: How to handle schema drift?

eldpbi — Tue, 03 Sep 2024 03:23:20 GMT

So Consider this scenario: Copying multiple csv files from OneLake to a Data warehouse within Fabric capacity. CSVs have different schemas, and the tables in the Data Warehouse have to be autocreated (will require schema drift to be on). Now since the schemas are different, in the "Mapping" section of the "Copy" activity, one cannot provide the schemas, but still the current pipelines in Fabric do ask for that.

For more context - GetMetadata gets all the childitems in the onelake folder, and ForEach runs the CopyData activity on all the ChildItems.

ajarora GraceGu haha