We have had several occurences where a Notebook will exit the notebook code, but the Spark session is inexplicably not closed.
This causes us to incur signifigant capacity consumption until an admin recognizes that a notebook has been running for much longer than expected via the Monitor tool or from Capacity alerts and manually ends the session. However, by the time corrective action has been taken, we have already utilized more capacity than we should have, leading to further potential issues like throttling.
Please add the ability to add an optional timeout for Notebooks that are run in the background. Once the time limit has been reached, the notebook should terminate and the spark session should be ended.
No CommentsBe the first to comment
Recent ideas
OneLake Catalog full-text and content search
Why is it not possible to search OneLake by item description or even content? So I was thinking about how a user can find a table if they don't know in which lakehouse it is located? Why can't you se...TePe15 hours agoHelper IVPlanned1.1KViews21likes5Comments