How to perform a MERGE using Spark SQL without listing every column or using "blind updates

Hi lavginqo3 ,
Thanks for reaching fabric community. I am happy to respond you and writing to fix issue you are facing.

This is a Spark SQL parse-time limitation. When Spark compiles a MERGE statement, it processes the WHEN MATCHED condition before it resolves the schemas of the tables involved. At that point, t.* is just a raw token. Spark doesn't yet know what columns t contains, so it cannot expand the wildcard. The engine throws the unresolved expression error and aborts.

Note: hash(s.*) works fine inside a subquery or a regular SELECT the restriction is specific to the MERGE condition clause.

The fix PySpark dynamic hash (no column listing required)

This generates the condition string dynamically so you never manually list columns. maintenance-free even as the schema evolves.

The solution is to let Python expand the columns before the SQL string is ever sent to Spark. By the time Spark sees the query, every column is named explicitly no wildcards, no error. You do need to list each column explicitly.

Example:

from pyspark.sql import SparkSession

spark = SparkSession.builder.getOrCreate()

# Step 1: fetch column list from schema — auto-updates if columns change
cols = [c for c in spark.table("bronze").columns if c != "id"]

# Step 2: build hash expressions for both sides
source_hash = f"hash({', '.join(f's.{c}' for c in cols)})"
target_hash = f"hash({', '.join(f't.{c}' for c in cols)})"

# Step 3: generate and run the MERGE
merge_sql = f"""
MERGE INTO silver AS t
USING bronze AS s
ON t.id = s.id
WHEN MATCHED AND {source_hash} != {target_hash}
  THEN UPDATE SET *
WHEN NOT MATCHED BY TARGET THEN INSERT *
WHEN NOT MATCHED BY SOURCE THEN DELETE
"""

spark.sql(merge_sql)

What Spark actually receives is already fully explicit. e.g. hash(s.name, s.age, s.city, ...) != hash(t.name, t.age, t.city, ...) so there are no wildcards to resolve and no error is thrown.

This is the cleanest approach you get full SQL MERGE semantics with zero column maintenance. The hash covers all non-key columns automatically.

Why this is maintenance-free

spark.table("bronze").columns always reflects the live schema. If you add, remove, or rename columns in your Bronze table, the hash condition updates automatically on the next run. you never touch this script.

If you find this response helpful, kindly consider marking it as the accepted solution and giving it a kudos. This helps others facing similar issues and is greatly appreciated.