Forum Discussion

GammaRamma's avatar
GammaRamma
Helper I
1 year ago

Cannot overwrite tables with copy activity when change data feed is enabled

The problem is with lakehouse tables on which I have enabled this:   spark.sql(f"ALTER TABLE {rawLkhParam}.{tableSchemaParam}.{tableNameParam} SET TBLPROPERTIES ('delta.enableChangeDataFeed' = '...
Pipeline
Anonymous's avatar
Anonymous
Not applicable
1 year ago

Hi GammaRamma ,

 

One way to do this is to use the .mode(“overwrite”) and .option(“overwriteSchema”, “true”) options when writing data to the table. This method allows you to update the table schema without deleting the table, thus preserving incremental history.

Such as below code:

dataframe.write \
    .format("delta") \
    .mode("overwrite") \
    .option("overwriteSchema", "true") \
    .partitionBy(<your-partition-columns>) \
    .saveAsTable("<your-table>")

 

Best Regards,
Adamk Kong

 

If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.

GammaRamma's avatar
GammaRamma
Helper I
1 year ago

Hello Anonymous, thank you for your response.

I have not tested this solution, but even if it does work, it doesn't really solve my problem the way I want. Using your method, I would have to create another table temporarily from my on prem source and then use a notebook to load the temp table as a dataframe and overwrite my original table. 

In a real bind this could be useful, but it's really not very efficient for a design solution. I think this is a problem with the copy activity which should be adressed by Microsoft.

Thanks anyway!