Forum Discussion

adamlob's avatar
adamlob
Icon for Advocate II rankAdvocate II
10 months ago
Solved

Hash Function for Row Compare

Hi,   I'm working on a Data Pipeline that loads data into a Dataverse table. I do a row compare to detect changes between loads, so I am only loading rows that have changed.   Is there anyway to ...
data engineering
  • AntoineW's avatar
    AntoineW
    10 months ago

    Hi adamlob,

     

    In Dataflow Gen2 / Fabric Data Pipeline (Data Factory), there’s no native “Hash” transformation yet.

    You can use Notebooks to do that : 

     

    from pyspark.sql.functions import sha2, concat_ws

    df_hashed = df.withColumn(
    "row_hash",
    sha2(concat_ws("|", *df.columns), 256)
    )

     

    Then save it back to your Lakehouse table and use that hash for change-detection.

    ✅ Benefits:

    • Very fast and scalable,

    • Produces fixed-length SHA-256 strings (~64 chars),

    • Easy to use as a comparison key.

    Doc : 

    - https://spark.apache.org/docs/latest/api/sql/index.html#sha2

     

     

    Hope it can help you !

    Best regards,

    Antoine

AntoineW's avatar
AntoineW
Icon for Super User rankSuper User
10 months ago

Hi adamlob,

 

In Dataflow Gen2 / Fabric Data Pipeline (Data Factory), there’s no native “Hash” transformation yet.

You can use Notebooks to do that : 

 

from pyspark.sql.functions import sha2, concat_ws

df_hashed = df.withColumn(
"row_hash",
sha2(concat_ws("|", *df.columns), 256)
)

 

Then save it back to your Lakehouse table and use that hash for change-detection.

✅ Benefits:

  • Very fast and scalable,

  • Produces fixed-length SHA-256 strings (~64 chars),

  • Easy to use as a comparison key.

Doc : 

- https://spark.apache.org/docs/latest/api/sql/index.html#sha2

 

 

Hope it can help you !

Best regards,

Antoine