Forum Discussion
Advocate IHow to handle schema drift?
In legacy datafactory there were options to explicitly allow schema drift. I do not see that in Fabric. Am I missing it?
For example,
- I was just doing some random testing of loading historical blob text files into a 'table' in Fabric to kick the tires
- I had created some intermediary parquet files from the text files to get practice with that
- I then attempted to load that parquet files to a 'table', and got the error..'Source column is not defined in delta meta data'
- Obviously because files have columns that evolved over time
I don't see a straight-forward way to handle this.
8 Replies
ajaroraMicrosoft Employee
How do you expect the schema variation to have taken effect ?
There can be many situations possible, column added, column dropped, or column type changed.
dhorrallProbably all of the above. Current 'datafactory' has checkbox to handle. I see nothing like this in Fabric. This was the basis of my question.
- ajarora
What you are referring to is possible through Azure Data Factory Mapping Dataflows, but they are not available in Fabric. Perhaps you want to try out Fabric Dataflows and see if it applies as it is ?
In terms of what copy activity allows, if your destination table already exists, and the data you are writing has a column missing, it will be defaulted to null (default value) when writing to destination. If there is a new column, or if a column is not typecastable to the destination type, then this is treated as a bad row, and you can either skip writing this bad row (and log it into a temporary storage to be processed later), or fail the operation (the default).
Any update on if schema drift will be added in the futre? especially being it looks like that functionality exsists in ADF?
So Consider this scenario: Copying multiple csv files from OneLake to a Data warehouse within Fabric capacity. CSVs have different schemas, and the tables in the Data Warehouse have to be autocreated (will require schema drift to be on). Now since the schemas are different, in the "Mapping" section of the "Copy" activity, one cannot provide the schemas, but still the current pipelines in Fabric do ask for that.
For more context - GetMetadata gets all the childitems in the onelake folder, and ForEach runs the CopyData activity on all the ChildItems.
