Forum Discussion
Solved
Notebook Merge Only Inserting Not Updating Matched Rows
See my code below: from delta.tables import *
# Get the target Delta table.
target_table = DeltaTable.forName(spark, "targettable")
# Define the source DataFrame (e.g., from a CSV file or ano...
Hi gdb729 ,
It sounds like you're really close but what you're describing usually points to the merge condition not evaluating as a true match, even though the data looks identical at first glance. A few things that can trip this up.
- Data type mismatches: For example, if target.id is an int and source.id is a string, Spark won't match them even if the values appear the same.
- Null values: Regular = comparison fails when either side is null. In Spark, null = null returns false. You’ll want to use the null-safe equality operator (<=>) instead.
- Whitespace / case issues: Especially on strings like transaction, even a trailing space can cause a mismatch.
Regards,
Akhil.
Hi gdb729 ,
It sounds like you're really close but what you're describing usually points to the merge condition not evaluating as a true match, even though the data looks identical at first glance. A few things that can trip this up.
- Data type mismatches: For example, if target.id is an int and source.id is a string, Spark won't match them even if the values appear the same.
- Null values: Regular = comparison fails when either side is null. In Spark, null = null returns false. You’ll want to use the null-safe equality operator (<=>) instead.
- Whitespace / case issues: Especially on strings like transaction, even a trailing space can cause a mismatch.
Regards,
Akhil.
Thanks for the quick response. Ended up being a null filter and when I swapped the equality operator which I didn't think I needed as that field shouldn't be null, merge worked correctly.