Schema mismatch error In Warehouse

[BhanuPrakash_K]

Goal:

Develop an end-to-end automated pipeline to extract data from an external source, transfer it to Microsoft Fabric Lakehouse,
perform data cleaning using a pre-designed PySpark notebook, save the cleaned data in a Lakehouse, and subsequently update
a data warehouse with the latest and cleaned data.

Implemented Steps:

1.Created a pipeline with a copy data activity to transfer data from Azure Blob to Fabric Lakehouse.
2.Integrated a notebook activity in the pipeline to execute a pre-designed PySpark notebook for data cleaning.
3.Added a copy data activity to move the cleaned data from the Lakehouse to the data warehouse for further data modeling.

Problems and Errors:
During the second run of the pipeline, schema mismatch errors occur when changes are made to the schema using the PySpark notebook.

Questions:

1.Is there any way to achive the automated pipeline even after the schema changes in a second run?
2.Does Microsoft Fabric Spports Dynamic Schema Handling and Schema Evaluation?
if possible give me reference links.

Thank You.

[Anonymous · Accepted solution]

Hi @BhanuPrakash_K ,

Thanks for using Fabric Community.

At present schema drift is not available in Fabric.
Copy activity allows, if your destination table already exists, and the data you are writing has a column missing, it will be defaulted to null (default value) when writing to destination. If there is a new column, or if a column is not typecastable to the destination type, then this is treated as a bad row, and you can either skip writing this bad row (and log it into a temporary storage to be processed later), or fail the operation (the default).

[BhanuPrakash_K]

Hi @Anonymous 
Thanks for your responce.

I recently intracted with Microsoft Development team regarding this issue. they concluded that the Schema drift and Auto refresh options are not available in Microsoft Fabric. 

[Anonymous]

Hi @BhanuPrakash_K ,

Glad to know that your query got resolved.
Please continue using Fabric Community for your further queries.

[Anonymous]

Hi @BhanuPrakash_K ,

We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet .
In case if you have any resolution please do share that same with the community as it can be helpful to others .
Otherwise, will respond back with the more details and we will try to help .

[Anonymous]

Hi @BhanuPrakash_K ,

We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet .
In case if you have any resolution please do share that same with the community as it can be helpful to others .
Otherwise, will respond back with the more details and we will try to help .