Forum Discussion
SessionExpiredException occurs regularly in driver's stderr (org.apache.zookeeper.ClientCnxn)
- 10 months ago
Hi dbeavon3,
Thank you for the response. In Fabric the spark coordination components(including ZooKeeper) are service-managed and containerized, not customer-managed.
They are isolated at the capacity level, not at the individual workspace level.- On a dedicated capacity, these services run within resources allocated only to that capacity and are not shared with other tenants.
- Multiple workspaces under the same capacity will share that capacity’s resources, but the service manages scheduling and isolation internally to prevent one job from affecting another.
- On a shared capacity, the coordination layer is multi-tenant and managed by Microsoft’s service fabric layer, but each job runs in its own isolated session.
You are right that these logs can appear even though the component is not customer-managed, they simply reflect transient coordination retries within the platform.
If the timeout changes reduce failures, it confirms a transient condition, If not the Fabric Support can review the backend ZooKeeper health for your job timestamps.
Thanks and regards,
Anjan Kumar Chippa
Hi dbeavon3
Hope you are fine. As per you, your jobs don't respond well and driver/executor is failing. Could you please tell me how you are using bronze, silver and gold layer. Also, In Databricks it is relatively easy and Microsoft adopted the same system (Delta Lake). In Microsoft Fabric, there is not different cluster ( job cluster, or all purpose cluster ) etc.
by the way, It's a Linux System. ( %fs ls ) commands.
Yes, we extract data with multiple layers.
Can you please focus on the technical reason for these job failures? The messages in the logs are meaningless without having a very low-level understanding of how Microsoft is hosting the so-called "ClientCnxn" (for zookeeper). I have NEVER seen zookeeper errors from my spark jobs on other platforms. And I have been working with spark for over three years now.