Forum Discussion
Schema mismatch error In Warehouse
Goal:
Develop an end-to-end automated pipeline to extract data from an external source, transfer it to Microsoft Fabric Lakehouse,
perform data cleaning using a pre-designed PySpark notebook, save the cleaned data in a Lakehouse, and subsequently update
a data warehouse with the latest and cleaned data.
Implemented Steps:
1.Created a pipeline with a copy data activity to transfer data from Azure Blob to Fabric Lakehouse.
2.Integrated a notebook activity in the pipeline to execute a pre-designed PySpark notebook for data cleaning.
3.Added a copy data activity to move the cleaned data from the Lakehouse to the data warehouse for further data modeling.
Problems and Errors:
During the second run of the pipeline, schema mismatch errors occur when changes are made to the schema using the PySpark notebook.
Questions:
1.Is there any way to achive the automated pipeline even after the schema changes in a second run?
2.Does Microsoft Fabric Spports Dynamic Schema Handling and Schema Evaluation?
if possible give me reference links.
Thank You.
Hi Anonymous ,
Thanks for using Fabric Community.
At present schema drift is not available in Fabric.
Copy activity allows, if your destination table already exists, and the data you are writing has a column missing, it will be defaulted to null (default value) when writing to destination. If there is a new column, or if a column is not typecastable to the destination type, then this is treated as a bad row, and you can either skip writing this bad row (and log it into a temporary storage to be processed later), or fail the operation (the default).
5 Replies
Hi Anonymous ,
Thanks for using Fabric Community.
At present schema drift is not available in Fabric.
Copy activity allows, if your destination table already exists, and the data you are writing has a column missing, it will be defaulted to null (default value) when writing to destination. If there is a new column, or if a column is not typecastable to the destination type, then this is treated as a bad row, and you can either skip writing this bad row (and log it into a temporary storage to be processed later), or fail the operation (the default).Hi Anonymous ,
We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet .
In case if you have any resolution please do share that same with the community as it can be helpful to others .
Otherwise, will respond back with the more details and we will try to help .Hi Anonymous ,
We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet .
In case if you have any resolution please do share that same with the community as it can be helpful to others .
Otherwise, will respond back with the more details and we will try to help .