Note book Session - Disconnects while running

Hi nagakosuri ,
The behavior described matches how notebook platforms track session activity compared to how Spark runs distributed tasks. Session timeouts usually depend on activity detected on the driver, like running cells, generating output, or sending heartbeat signals to the control plane. However, long running Spark tasks such as large Delta Lake writes or blocking I/O when reading from SharePoint, mainly happen on executor nodes or leave the driver waiting without visible activity. As a result, the driver may look idle to the control plane, even though computation is still happening in the background. If this apparent inactivity lasts longer than the set timeout, the platform might label the session as inactive and end it. So, the disconnection isn’t caused by a Spark job failure, but by the inactivity detection relying on driver-side signals that don’t always show ongoing work.