topic Get notebook_id of running notebook in high concurrency in Data Science

Get notebook_id of running notebook in high concurrency

KimTutein — Thu, 16 Jan 2025 10:51:11 GMT

Hi Community

I have a small issue I hope someone may guide me to a solution. I simply want to know the notebook_id of the running notebook (without hardcoding the name of the notebook). I have used the function get_artifact_id() from sempy.fabric library. This will give the notebook_id – however if the notebook is joining a high concurrency session you will get the notebook_id of the notebook that started the session and not the one that is running.

If anyone have some input it will be much appreciated.

Regards

Kim Tutein

Appendix – example will work when running in separate session but not if joining high concurrency session

from sempy import fabric

notebook_id = fabric.get_artifact_id() #notebook id

print(notebook_id)

Re: Get notebook_id of running notebook in high concurrency

nilendraFabric — Thu, 16 Jan 2025 12:51:11 GMT

KimTutein

In a high-concurrency session in Microsoft Fabric, obtaining the notebook_id of the currently running notebook can be challenging because the fabric.get_artifact_id() function from the SemPy library retrieves the ID of the notebook that initiated the shared Spark session, rather than the specific notebook currently running. This behavior is due to how high-concurrency sessions are designed to share a single Spark context across multiple notebooks, improving efficiency and reducing costs.

If it is possible, avoid joining a high-concurrency session and instead run the notebook in its own Spark session. This ensures that fabric.get_artifact_id() returns the correct notebook_id for the running notebook.

else implement logging mechanisms where each notebook logs its execution details (such as its name or purpose) into a shared storage or monitoring system. This can help identify which notebook is running without relying solely on notebok_id.

I havent explored it yet but worth exploring

Session Metadata or Tagging:High-concurrency sessions in Fabric allow tagging of sessions or notebooks for identification. However, there is no direct API to dynamically fetch the specific notebook_id within such sessions. You may consider using custom tags or metadata to track notebooks manually.

If this post helps, then please give us Kudos and consider Accept it as a solution to help the other members find it more quickly.

Re: Get notebook_id of running notebook in high concurrency

KimTutein — Thu, 16 Jan 2025 13:07:13 GMT

Hi nilendraFabric

Thank you for your answer. For now we are simple not running notebooks in high concurrency. As we are using private endpoint we have to wait 3-6 minutes for every startup. As we use the notebook_id in our logging it is very important for us. For now I guess there is not solution to get the correct notebook_id other than hardcode the name of the notebook and lookup the id looping thorugh the items in the workspace. However this is not a very safe way of doint this as if the notebook change name it will no longer find the correct workspace_id

Re: Get notebook_id of running notebook in high concurrency

Anonymous — Tue, 21 Jan 2025 04:28:59 GMT

Hi KimTutein
Thank you for reaching out microsoft fabric community forum.
I tried to look for alternate solution in Sempy library and it seems your conclusion is correct but i would suggest to once tryprint(notebookutils.runtime.context['currentNotebookId']) and check how it behaves for private endpoint.

If you need any further assistance or have any questions, please feel free to reach us.

If this solution helps, please consider giving us Kudos and accepting it as the solution so that it may assist other members in the community.

Thanks and Regards.

Re: Get notebook_id of running notebook in high concurrency

KimTutein — Tue, 21 Jan 2025 06:17:17 GMT

Hi Anonymous

I allso tried looking all around the Sempy library without luck. However I tried your solution with notebookutills and that works fine -- also with privete endpoint 🙂 :-).

This is very important for us if running in high concurrency so thank you very much for your help and guidance.

Regards

Kim Tutein