Forum Discussion
Spark Silent Failure
Thank you very much for the response. I tried running again and it "failed". Then as per your recommendation I configured this:
#Based on comments from the fabric community - try these until Microsoft resolves this issuespark.conf.set("spark.sql.shuffle.partitions", "10")spark.conf.set("spark.sql.adaptive.enabled", "false")#spark.conf.set("spark.serializer", "org.apache.spark.serializer.KryoSerializer") - not exposed in Fabricspark.conf.set("spark.sql.autoBroadcastJoinThreshold", "-1")This works for us, and restores our deduplication process for the time being until Microsoft fixes the regression issue.
We actually ran into a very similar issue recently, so your post sounds familiar. Our Spark jobs had been working fine for months, then suddenly one phase started “succeeding” with no output and logs stopping halfway through. The Spark UI also showed “busy” and wouldn’t load while the job was running.
In our case, nothing in the code or data changed — the only thing that lined up was a recent Fabric runtime/VHD update. After that update, one part of the workflow would hang quietly instead of throwing an error.
A few things that helped us narrow it down:
Running the same notebook in a new workspace worked, which told us it wasn’t the logic.
Clearing the session and re-attaching the lakehouse made the job run further, but not consistently.
Re-adding our dependency jars forced the environment to refresh and helped temporarily.
Not perfect fixes, but enough to confirm it wasn’t data size or Zingg config — it was the environment.
Since Microsoft already told you the VHD changed, your support case is definitely the right path. If they share a workaround or rollback option, please update here — this might be affecting more people.
GopiKrishna