Forum Discussion
upsert in lakehouse table using python not pyspark
I am using the below code and it is working,
if os.path.exists("/lakehouse/default/Tables/test_pandas") :dt = DeltaTable("/lakehouse/default/Tables/test_pandas",storage_options={"allow_unsafe_rename": "true"})(dt.merge(source=expanded_df,predicate="s.row_hash = t.row_hash",source_alias="s",target_alias="t",).when_matched_update_all().when_not_matched_insert_all().execute())print("Table updated : test_pandas")else:write_deltalake(table_path, expanded_df, mode = 'overwrite', schema_mode='merge', engine = 'rust', storage_options=storage_options)print("Table created for the first time")Thanks
Community SupportHi fabric_1 ,
Thank you for reaching out to us on the Microsoft Fabric Community Forum.
You are correct — the write_deltalake() function, which utilizes the delta-rs engine (engine='rust'), does not currently support native MERGE INTO or upsert operations as PySpark does. It only allows basic write modes such as 'append', 'overwrite', and 'error'.
However, it is possible to simulate upsert behavior as a workaround using pure Python. This typically involves manually merging the new data with the existing data and handling deduplication before writing it back to the table. Please note that this approach lacks transactional guarantees. As you mentioned that you used small dataset once try this.
If this post was helpful, please give us Kudos and consider marking Accept as solution to assist other members in finding it more easily.