Forum Discussion
Question about internal bugs in "notebookUtils" "mount operation" for default lakehouse
I'm about ready to open my tenth bug in Fabric-Spark. They are taking me over 10+ hours just to transmit these to the PG and get them to engage.... . So I'm hoping someone has seen this error and I won't have to spend the next two weeks waiting on that Fabric-Spark team:
A pyspark notebook started executing cells today at 8:29:57 AM utc:
Not long after that it dies, when trying to open a file on LH:
... the error is pretty meaningless, as you can see above. A file that is in the LH cannot be read, and there is no good reason. This only affected one notebook execution out of about ~30 that were performed this morning.
After I review the "stderr" of the driver, it appears that this notebook session was doomed for failure before it even started. A "mount operation" had failed but had not prevented the execution of the notebook. This took place at 8:29:25 AM utc.
...
2025-01-22 08:29:25,506 INFO notebookUtils [pool-52-thread-1]: [mount operation] mount point /default/Files info has been updated with operationId 10c731d8-bdf8-4a86-a977-926f9d9cf7ee
...
2025-01-22 08:29:55,625 INFO notebookUtils [pool-52-thread-1]: [checkDriverNodeMountStatus] operationId: 10c731d8-bdf8-4a86-a977-926f9d9cf7ee check driver node cost 30116ms, key:lakehousemounts_vm-ae003372_/default/Files/, result: timeout
2025-01-22 08:29:55,627 INFO notebookUtils [pool-52-thread-1]: Rollback the operation 10c731d8-bdf8-4a86-a977-926f9d9cf7ee due to mount /default/Files failed.
...
2025-01-22 08:29:56,632 ERROR UserConsole [pool-52-thread-1]: java.util.concurrent.RejectedExecutionException: Task scala.concurrent.impl.CallbackRunnable@4960f147 rejected from java.util.concurrent.ThreadPoolExecutor@4de8e398[Shutting down, pool size = 2, active threads = 2, queued tasks = 0, completed tasks = 0]
2025-01-22 08:29:56,633 ERROR UserConsole [pool-52-thread-1]:
You can see that as-of 8:29:56 AM utc, this pyspark session was already falling over. Yet the notebook cells proceeded to start to be executed despite the problems. Eventually the notebook dies once we reach the part of the custom code where we expect the LH files to be available for use.
Has anyone already encountered this? Is it on a "known issues" list?
Does this product expect the notebook users to perform validation at the start of our notebooks to ensure that the session/environment isn't corrupted from the very beginning? It is not self-evident that customers in Fabric should do this duplicate work to verify the health of the session.
Is there a notebook util method we can execute, to ensure the session is usable?
Any help would be appreciated.
Hi dbeavon3 ,
Thank you for bringing this issue to our attention. We have investigated the matter and verified that it is not a known issue at this time.
We apologize for the inconvenience and understand your frustration. Unfortunately, we cannot escalate this to the support team directly.
If you still face the issue then please consider raising a support ticket for further assistance.
To raise a support ticket for Fabric and Power BI, kindly follow the steps outlined in the following guide:
How to create a Fabric and Power BI Support ticket - Power BI | Microsoft Learn
Thankyou for your Paitence and understanding.
7 Replies
I see now that there are multiple threads failing in the stderr of the driver.
At first I was only focused on pool-52-thread-1
... it is really unfortunate that the Fabric team allows notebook cells to start executing despite the severity of problems in the session itself. That only leads to massive confusion, and finger-pointing since the notebook ultimately dies in a custom code cell, yet this death is triggered by a problem that was put into motion a minute prior. (back within the code that belongs to Microsoft )
Hopefully the PG will start working on this soon, and add it to their known issues list for the sake of the community.
Hi dbeavon3 ,
Thank you for bringing this issue to our attention. We have investigated the matter and verified that it is not a known issue at this time.
We apologize for the inconvenience and understand your frustration. Unfortunately, we cannot escalate this to the support team directly.
If you still face the issue then please consider raising a support ticket for further assistance.
To raise a support ticket for Fabric and Power BI, kindly follow the steps outlined in the following guide:
How to create a Fabric and Power BI Support ticket - Power BI | Microsoft Learn
Thankyou for your Paitence and understanding.
Hi V-yubandi-msft
Thanks for the reply.
Do you mean it is not known to customers, or is not known to Mindtree (ICM's/SR's), or is not known to this Microsoft PG, or has not been published to the "known issues" list?I am not sure I know what you mean, but I will go ahead and give you the benefit of the doubt. The ticket is reported here:
TrackingID#2501220040016849.
... but I was under the impression that I'm (at least) the tenth customer to report this bug. It seems extremely unlikely that these file-system mounting bugs are not known to the PG. In fact, it appears they already implemented 3 retries to try to avoid the bug:
In my experience the various PG's of Fabric will typically expect their customers to be the ones who own responsibilty for implementing all these "retries". But when a PG agrees to perform these retries in their own code, then you can be 100% certain that they are very familiar with the underlying bugs.
I also want to point out how unexpectedly this issue can arise. Out of thirty notebooks running in a loop, one of them might have a mounting issue like so:
Side note: I want to put these message here once more to make sure they appear in search results:2025-01-22 08:29:55,625 INFO notebookUtils [pool-52-thread-1]: [checkDriverNodeMountStatus] operationId: 10c731d8-bdf8-4a86-a977-926f9d9cf7ee check driver node cost 30116ms, key:lakehousemounts_vm-ae003372_/default/Files/, result: timeout 2025-01-22 08:29:55,627 INFO notebookUtils [pool-52-thread-1]: Rollback the operation 10c731d8-bdf8-4a86-a977-926f9d9cf7ee due to mount /default/Files failed. 2025-01-22 08:29:55,736 INFO notebookUtils [pool-52-thread-2]: [checkDriverNodeMountStatus] operationId: a42ced8c-b6df-4623-8ab5-80c6b579ed2a check driver node cost 30103ms, key:lakehousemounts_vm-ae003372_/default/Tables/, result: timeout 2025-01-22 08:29:55,737 INFO notebookUtils [pool-52-thread-2]: Rollback the operation a42ced8c-b6df-4623-8ab5-80c6b579ed2a due to mount /default/Tables failed.
