Forum Discussion
Solution SageCOUNT is 0 when nullable column in where clause?
There is a NULL-safe equality operator in SparkSQL <=>
This properly handles NULLs and equality and can replace the need for coalesces.
https://spark.apache.org/docs/3.5.2/sql-ref-null-semantics.htmlA solution is not present in this thread, despite someone accepting a reply as a solution.
HI jeffshieldsdev,
Perhaps you can take a look at following link about use sql statement to check null values if help with your scenario:
t sql - How to check if a column is empty or null using SQL query select statement? - Stack Overflow
Regards,
Xiaoxin Sheng
Solution SageI'm not struggling with null values, I wanted to confirm this strange behavior in Spark SQL in Fabric: one is equals, the other is doesn't equal--both are 0
select count(*)
from my_table
where internal_id = external_id;
-- 0
select count(*)
from my_table
where internal_id <> external_id;
-- 0
HI jeffshieldsdev,
Have you tried to manually specify the records that has same value and use them to run with your sql queries?
Also, I think you need to check these records at first to confirm and apply upper/lower function on these column to avoid case sensitive, do trim/remove 'blank space' and special characters to help them mapping correctly.Regards,
Xiaoxn Sheng