Forum Discussion
Advocate IFabric Notebook Stuck in "Running" Status
Hi smoqt,
Thank you for your follow-up and the update provided.
-
At present, I have not found any documentation to confirm that this is a bug.The underlying cause seems to be related to improper token caching logic during Spark session initialization, as well as missing runtime dependencies (such as ZooKeeper classes) in the driver environment.
-
Pipeline activity timeouts do not terminate the associated Spark job. Consequently, the notebook may continue to run in the background, consuming Capacity Units (CUs). To manage this, we recommend using the "Monitor Activities" feature in Microsoft Fabric, as detailed in the Monitor activities in Microsoft Fabric - Training | Microsoft Learn, to track and manually terminate long-running jobs. Additionally, you may consider implementing a watchdog script leveraging the REST API to monitor and cancel jobs exceeding a predefined runtime.
-
Please utilise the Capacity Metrics app to identify notebooks that have high CU consumption. It is advisable to break lengthy notebooks into smaller, modular units to minimise the risk of idle drivers. Furthermore, configuring the Spark job admission rules to limit concurrency may help prevent system overload.
Given the systemic nature of this issue and the potential cost implications, we recommend raising a support ticket with Microsoft for a thorough investigation and resolution. You may raise a ticket via the Microsoft Fabric Support and Status | Microsoft Fabric.
Additionally, please refer to the following links:
What is the Microsoft Fabric Capacity Metrics app? - Microsoft Fabric | Microsoft LearnUnderstand the metrics app compute page - Microsoft Fabric | Microsoft Learn
Install the Microsoft Fabric capacity metrics app - Microsoft Fabric | Microsoft LearnIf you find our response helpful, kindly mark it as the accepted solution. This will assist other community members facing similar queries.
Should you have any further questions, please feel free to reach out to the Microsoft Fabric community.
Thank you.
-
Community SupportHi smoqt,
Thank you for your follow-up and the update provided.
-
At present, I have not found any documentation to confirm that this is a bug.The underlying cause seems to be related to improper token caching logic during Spark session initialization, as well as missing runtime dependencies (such as ZooKeeper classes) in the driver environment.
-
Pipeline activity timeouts do not terminate the associated Spark job. Consequently, the notebook may continue to run in the background, consuming Capacity Units (CUs). To manage this, we recommend using the "Monitor Activities" feature in Microsoft Fabric, as detailed in the Monitor activities in Microsoft Fabric - Training | Microsoft Learn, to track and manually terminate long-running jobs. Additionally, you may consider implementing a watchdog script leveraging the REST API to monitor and cancel jobs exceeding a predefined runtime.
-
Please utilise the Capacity Metrics app to identify notebooks that have high CU consumption. It is advisable to break lengthy notebooks into smaller, modular units to minimise the risk of idle drivers. Furthermore, configuring the Spark job admission rules to limit concurrency may help prevent system overload.
Given the systemic nature of this issue and the potential cost implications, we recommend raising a support ticket with Microsoft for a thorough investigation and resolution. You may raise a ticket via the Microsoft Fabric Support and Status | Microsoft Fabric.
Additionally, please refer to the following links:
What is the Microsoft Fabric Capacity Metrics app? - Microsoft Fabric | Microsoft Learn
Understand the metrics app compute page - Microsoft Fabric | Microsoft Learn
Install the Microsoft Fabric capacity metrics app - Microsoft Fabric | Microsoft Learn
If you find our response helpful, kindly mark it as the accepted solution. This will assist other community members facing similar queries.
Should you have any further questions, please feel free to reach out to the Microsoft Fabric community.
Thank you.
smoqtI will raise a support ticket. Thank you.