Forum Discussion
Memorable MemberSessionExpiredException occurs regularly in driver's stderr (org.apache.zookeeper.ClientCnxn)
Hi dbeavon3,
Thank you for the response. In Fabric the spark coordination components(including ZooKeeper) are service-managed and containerized, not customer-managed.
They are isolated at the capacity level, not at the individual workspace level.- On a dedicated capacity, these services run within resources allocated only to that capacity and are not shared with other tenants.
- Multiple workspaces under the same capacity will share that capacity’s resources, but the service manages scheduling and isolation internally to prevent one job from affecting another.
- On a shared capacity, the coordination layer is multi-tenant and managed by Microsoft’s service fabric layer, but each job runs in its own isolated session.
You are right that these logs can appear even though the component is not customer-managed, they simply reflect transient coordination retries within the platform.
If the timeout changes reduce failures, it confirms a transient condition, If not the Fabric Support can review the backend ZooKeeper health for your job timestamps.
Thanks and regards,
Anjan Kumar Chippa
Memorable Member>> "managed within the service".
Ideally we wouldn't be seeing errors about these things that we don't manage ourselves; especially when they don't have any customer-facing surface area and we cannot troubleshoot them.
I will try the configuration settings that you shared and see if there is a difference.
>> If your workspace runs on dedicated capacity, those resources are isolated to your tenant
Are they isolated to a workspace? Is it possible that my workloads are conflicting with other workloads in the same tenant? It is unfortunate that the zookeeper behavior is decreasing the reliability and health of the cluster. I'm guessing it is underprovisioned in some way, or shared in ways that customers wouldn't expect.
Community SupportHi dbeavon3,
Thank you for the response. In Fabric the spark coordination components(including ZooKeeper) are service-managed and containerized, not customer-managed.
They are isolated at the capacity level, not at the individual workspace level.
- On a dedicated capacity, these services run within resources allocated only to that capacity and are not shared with other tenants.
- Multiple workspaces under the same capacity will share that capacity’s resources, but the service manages scheduling and isolation internally to prevent one job from affecting another.
- On a shared capacity, the coordination layer is multi-tenant and managed by Microsoft’s service fabric layer, but each job runs in its own isolated session.
You are right that these logs can appear even though the component is not customer-managed, they simply reflect transient coordination retries within the platform.
If the timeout changes reduce failures, it confirms a transient condition, If not the Fabric Support can review the backend ZooKeeper health for your job timestamps.
Thanks and regards,
Anjan Kumar Chippa