Spark Silent Failure

This is almost a Fabric runtime regression not your zingg configuration. When a workload runs perfectly for months then starts “succeeding” with no output, no errors, and blocked Spark UI access, it points to changes in the underlying Spark VHD or fabric worker image. Your support ticket note about “VHD appears to have changed” confirms that something in the Nov2024 rollout likely broke long-running or shuffle heavy jobs.

Key signals that this is platform-side, not Zingg side:

1. All earlier Zingg phases run fine - Only match fails. Match is the most shuffle-intensive stage and the first to expose regressions in cluster image or shuffle service.

2. Silent success with partial logs is a typical Fabric worker crash or executor reset. Fabric often marks these as “successful” if the driver exits cleanly.

3. Spark monitor being unavailable is a known symptoms when the driver is stuck in a dead state or the worker hosting the UI crashed.

4. Same failure happens in Synapse runtime as well. That strongly suggests that both runtimes now share the updated VHD image.

What you can try locally, while you wait for microsoft engineering:

1. Explicitly pin shuffle partitions
   spark.sql.shuffle.partitions = 50 (or even lower for 15k rows)

2. Disable adaptive execution
   spark.sql.adaptive.enabled = false

3. Force Kryo serialization
   spark.serializer = org.apache.spark.serializer.KryoSerializer

4. Reduce broadcast threshold
   spark.sql.autoBroadcastJoinThreshold = -1

5. Run match with very small sample to confirm the job actually enters Zingg’s matching phase. If even 500 rows fail silently, the problem is platform level.

6. Try running in a completely new workspace and new Lakehouse (clean compute root). This occasionally avoids corrupted mount artifacts.

realistically speaking, none of these are likely to fix a true VHD regression, but they help you rule out config issues and give repro data to support.