topic Re: Issue: Spark sessions running >39 hours despite 2-hour timeout in Data Engineering

Issue: Spark sessions running >39 hours despite 2-hour timeout

nguyenhieubis — Mon, 18 Aug 2025 02:45:28 GMT

I have a pipeline that runs 4 notebook tasks, using the High concurrency feature for running multiple notebooks. I set the task timeout to "0.02:00:00".

However, since August 1, 2025, I have encountered around 4–5 cases where the Spark session gets stuck in a running state (e.g., 39 hours) and does not force stop after the 2-hour timeout as expected.

Has anyone experienced the same issue, or is there a known workaround/fix?
Thank you in advance for your support and guidance!

Re: Issue: Spark sessions running >39 hours despite 2-hour timeout

BalajiL — Mon, 18 Aug 2025 05:21:41 GMT

nguyenhieubis : Looks it is a known issue in Fabric especially in high concurrency run time in spark. The task time out control only at orchestration layer to wait for a notebook to complete. It does not forcefully terminate the spark session. You can try to set the explicit timeout inside the notebook itself.

Thanks !

Re: Issue: Spark sessions running >39 hours despite 2-hour timeout

v-hashadapu — Mon, 25 Aug 2025 05:00:25 GMT

Hi nguyenhieubis , hope you are doing great. May we know if your issue is solved or if you are still experiencing difficulties. Please share the details as it will help the community, especially others with similar issues.

Re: Issue: Spark sessions running >39 hours despite 2-hour timeout

nguyenhieubis — Mon, 25 Aug 2025 06:13:47 GMT

Hi v-hashadapu , I'm still monitoring it. I don't have this issue anymore now, but I'm not sure if I will have it again. It's still worth monitoring, I know Fabric is still in development and improvement, so I hope this issue will be fixed to make it more stable.

Re: Issue: Spark sessions running >39 hours despite 2-hour timeout

v-hashadapu — Mon, 25 Aug 2025 06:25:56 GMT

Hi nguyenhieubis , Thanks for the sharing the information here. I hope the issue is cleared permanently for you. If you have any other queries, please feel free to raise a new post in the community. We are always happy to help.

Re: Issue: Spark sessions running >39 hours despite 2-hour timeout

v-hashadapu — Fri, 29 Aug 2025 09:06:16 GMT

Hi nguyenhieubis , hope you are doing well. Just wanted to check on your issue to see if it came back or if everything is going well. Please let us know.

Thank you.