Hash Function for Row Compare

adamlob — Thu, 23 Oct 2025 00:58:23 GMT

Hi,

I'm working on a Data Pipeline that loads data into a Dataverse table. I do a row compare to detect changes between loads, so I am only loading rows that have changed.

Is there anyway to hash the concat of rows? At the moment it seems I can only do plain-text and then convert it to Binary. Hashing would help save on space.

Re: Hash Function for Row Compare

AntoineW — Thu, 23 Oct 2025 08:16:44 GMT

Hi adamlob,

In Dataflow Gen2 / Fabric Data Pipeline (Data Factory), there’s no native “Hash” transformation yet.

You can use Notebooks to do that :

from pyspark.sql.functions import sha2, concat_ws

df_hashed = df.withColumn(
"row_hash",
sha2(concat_ws("|", *df.columns), 256)
)

Then save it back to your Lakehouse table and use that hash for change-detection.

✅ Benefits:

Very fast and scalable,
Produces fixed-length SHA-256 strings (~64 chars),
Easy to use as a comparison key.

Doc :

- https://spark.apache.org/docs/latest/api/sql/index.html#sha2

Hope it can help you !

Best regards,

Antoine

topic Hash Function for Row Compare in Data Engineering

Hash Function for Row Compare

Re: Hash Function for Row Compare