upsert in lakehouse table using python not pyspark

[fabric_1]

Hi,

I have a small dataset that I am using in my project and trying to perform upserts in a lakehouse table using python in fabric notebooks, but couldn't figure out a way to do that. I can overwrite or append the table using this

write_deltalake(table_path, expanded_df, mode = 'overwrite', schema_mode='merge', engine = 'rust', storage_options=storage_options)

I need to do perform upsert here, Is there a way to do using python not pyspark?

Thanks

[fabric_1 · Accepted solution]

 I am using the below code and it is working,

if os.path.exists("/lakehouse/default/Tables/test_pandas") :

dt = DeltaTable("/lakehouse/default/Tables/test_pandas",storage_options={"allow_unsafe_rename": "true"})

(

dt.merge(

source=expanded_df,

predicate="s.row_hash = t.row_hash",

source_alias="s",

target_alias="t",

)

.when_matched_update_all()

.when_not_matched_insert_all()

.execute()

)

print("Table updated : test_pandas")

else:

write_deltalake(table_path, expanded_df, mode = 'overwrite', schema_mode='merge', engine = 'rust', storage_options=storage_options)

print("Table created for the first time")

Thanks

[v-menakakota]

Hi @fabric_1 ,

We really appreciate your efforts and for letting us know the update on the issue.

Please continue using fabric community forum for your further assistance.

If this is the solution that has worked for you please accept your reply as solution so as to help other community members who may face similar issue in the future.
Thank you for reaching out to us on the Microsoft Fabric Community Forum.

[v-menakakota]

Hi @fabric_1 ,

Thank you for reaching out to us on the Microsoft Fabric Community Forum.

You are correct — the write_deltalake() function, which utilizes the delta-rs engine (engine='rust'), does not currently support native MERGE INTO or upsert operations as PySpark does. It only allows basic write modes such as 'append', 'overwrite', and 'error'.

However, it is possible to simulate upsert behavior as a workaround using pure Python. This typically involves manually merging the new data with the existing data and handling deduplication before writing it back to the table. Please note that this approach lacks transactional guarantees. As you mentioned that you used small dataset once try this.

If this post was helpful, please give us Kudos and consider marking Accept as solution to assist other members in finding it more easily.

[v-menakakota]

Hi @fabric_1 ,

May I ask if you have resolved this issue? If so, please mark the helpful reply and accept it as the solution. This will be helpful for other community members who have similar problems to solve it faster.

Thank you.

[v-menakakota]

Hi @fabric_1 ,

I wanted to check if you had the opportunity to review the information provided. Please feel free to contact us if you have any further questions. If my response has addressed your query, please accept it as a solution and give a 'Kudos' so other members can easily find it.

Thank you.

[v-menakakota]

Hi @fabric_1 ,

I hope this information is helpful. Please let me know if you have any further questions or if you'd like to discuss this further. If this answers your question, please Accept it as a solution and give it a 'Kudos' so others can find it easily.

Thank you.