Forum Discussion
Solved
Spark Silent Failure
Long time listener, first time caller here. We have our core deduplication process configured in Fabric. It had been running exceptional for months. The last succesful run was on November 13, the...
Thank you very much for the response. I tried running again and it "failed". Then as per your recommendation I configured this:
#Based on comments from the fabric community - try these until Microsoft resolves this issuespark.conf.set("spark.sql.shuffle.partitions", "10")spark.conf.set("spark.sql.adaptive.enabled", "false")#spark.conf.set("spark.serializer", "org.apache.spark.serializer.KryoSerializer") - not exposed in Fabricspark.conf.set("spark.sql.autoBroadcastJoinThreshold", "-1")This works for us, and restores our deduplication process for the time being until Microsoft fixes the regression issue.
Hi ps19234561 ,
Thank you for confirming the results. It’s great to hear that the configuration adjustments helped restore your deduplication process while the underlying regression is being addressed. If you run into any additional issues or have further questions as you continue testing, please feel free to reach out again in the Fabric Community we're here to support you.
Thnak you.