Forum Discussion

fabric_1's avatar
fabric_1
New Member
1 year ago
Solved

upsert in lakehouse table using python not pyspark

Hi, I have a small dataset that I am using in my project and trying to perform upserts in a lakehouse table using python in fabric notebooks, but couldn't figure out a way to do that. I can overwrit...
data engineering
lakehouse
notebook
  • fabric_1's avatar
    fabric_1
    1 year ago

     I am using the below code and it is working,

     

    if os.path.exists("/lakehouse/default/Tables/test_pandas") :
    dt = DeltaTable("/lakehouse/default/Tables/test_pandas",storage_options={"allow_unsafe_rename": "true"})
    (
    dt.merge(
    source=expanded_df,
    predicate="s.row_hash = t.row_hash",
    source_alias="s",
    target_alias="t",
    )
    .when_matched_update_all()
    .when_not_matched_insert_all()
    .execute()
     
    )
    print("Table updated : test_pandas")
    else:
    write_deltalake(table_path, expanded_df, mode = 'overwrite', schema_mode='merge', engine = 'rust', storage_options=storage_options)
    print("Table created for the first time")
     
    Thanks
v-menakakota's avatar
v-menakakota
Community Support
1 year ago

Hi fabric_1 ,

I wanted to check if you had the opportunity to review the information provided. Please feel free to contact us if you have any further questions. If my response has addressed your query, please accept it as a solution and give a 'Kudos' so other members can easily find it.


Thank you.

v-menakakota's avatar
v-menakakota
Community Support
1 year ago

Hi fabric_1 ,

I hope this information is helpful. Please let me know if you have any further questions or if you'd like to discuss this further. If this answers your question, please Accept it as a solution and give it a 'Kudos' so others can find it easily.

Thank you.