Forum Discussion
Spark Silent Failure
We actually ran into a very similar issue recently, so your post sounds familiar. Our Spark jobs had been working fine for months, then suddenly one phase started “succeeding” with no output and logs stopping halfway through. The Spark UI also showed “busy” and wouldn’t load while the job was running.
In our case, nothing in the code or data changed — the only thing that lined up was a recent Fabric runtime/VHD update. After that update, one part of the workflow would hang quietly instead of throwing an error.
A few things that helped us narrow it down:
Running the same notebook in a new workspace worked, which told us it wasn’t the logic.
Clearing the session and re-attaching the lakehouse made the job run further, but not consistently.
Re-adding our dependency jars forced the environment to refresh and helped temporarily.
Not perfect fixes, but enough to confirm it wasn’t data size or Zingg config — it was the environment.
Since Microsoft already told you the VHD changed, your support case is definitely the right path. If they share a workaround or rollback option, please update here — this might be affecting more people.
GopiKrishna