Forum Discussion

ps19234561's avatar
ps19234561
New Member
10 months ago
Solved

Spark Silent Failure

Long time listener, first time caller here.   We have our core deduplication process configured in Fabric. It had been running exceptional for months. The last succesful run was on November 13, the...
data engineering
  • ps19234561's avatar
    ps19234561
    10 months ago

    Thank you very much for the response. I tried running again and it "failed". Then as per your recommendation  I configured this:

     

    #Based on comments from the fabric community - try these until Microsoft resolves this issue
    spark.conf.set("spark.sql.shuffle.partitions", "10")
    spark.conf.set("spark.sql.adaptive.enabled", "false")
    #spark.conf.set("spark.serializer", "org.apache.spark.serializer.KryoSerializer") - not exposed in Fabric
    spark.conf.set("spark.sql.autoBroadcastJoinThreshold", "-1")
     
    This works for us, and restores our deduplication process for the time being until Microsoft fixes the regression issue. 
michaeldestein's avatar
michaeldestein
Regular Visitor
13 hours ago

Sorry you’re dealing with this one. A job that reports success while quietly writing nothing is about the most frustrating failure mode Spark has. (Disclosure: I work at Quantexa, which builds Quantexa Unify, a native Fabric workload for entity resolution.)

A couple of things might help narrow it down while you wait on Microsoft. Fabric applies maintenance updates to runtimes, so comparing the library versions in your environment before and after November 13 could show what moved underneath Zingg. Pointing the match output at a plain parquet path in Files for one test run would also tell you whether the problem sits in the match itself or in the final write.

If you’d like a longer-term option that takes the library and runtime pairing off your plate, Quantexa Unify is worth a look. It’s a native Fabric workload you add from the Workload Hub, so there’s no package version or Spark configuration for your team to manage. Unify runs on your own capacity inside your tenant, reads your tables straight from OneLake, proposes the field mappings for you and writes deduplicated entities back to your lakehouse as Delta tables, each linked to the source records behind it. You also skip the training and labeling phases entirely.

Your volume would be a light lift. Our testing processed over 5 million records per hour on an F2, so 15,000 rows would finish in a fraction of the run time you’re seeing now. The output is ready for Power BI, notebooks and Fabric IQ as soon as the run completes.

Happy to help you try it against your own data if that’s useful, and I hope the support case turns up a quick fix in the meantime.