Forum Discussion

GammaRamma's avatar
GammaRamma
Helper I
1 year ago

Cannot overwrite tables with copy activity when change data feed is enabled

The problem is with lakehouse tables on which I have enabled this:   spark.sql(f"ALTER TABLE {rawLkhParam}.{tableSchemaParam}.{tableNameParam} SET TBLPROPERTIES ('delta.enableChangeDataFeed' = '...
Pipeline
Anonymous's avatar
Anonymous
Not applicable
1 year ago

Hi GammaRamma ,

 

One way to do this is to use the .mode(“overwrite”) and .option(“overwriteSchema”, “true”) options when writing data to the table. This method allows you to update the table schema without deleting the table, thus preserving incremental history.

Such as below code:

dataframe.write \
    .format("delta") \
    .mode("overwrite") \
    .option("overwriteSchema", "true") \
    .partitionBy(<your-partition-columns>) \
    .saveAsTable("<your-table>")

 

Best Regards,
Adamk Kong

 

If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.