The ultimate Fabric, Power BI, SQL, and AI community-led learning event. Save €200 with code FABCOMM.
Get registeredEnhance your career with this limited time 50% discount on Fabric and Power BI exams. Ends August 31st. Request your voucher.
TLDR -
I am following this and that to install libraries. Once a library is installed, it works in the current notebook but I am unable to utilize it in any other. It seems the installed libraries are notebook-scoped than clusters-scoped. But I never had these sort of issues with requests, aiohttp and other libraries previously.
E.g. termcolor is installed through notebook3 and works as expected in notebook3
but fails in notebook4 executed shortly after
Solved! Go to Solution.
Hi @smpa01 ,
You're currect, using a custom environment does introduce performance overhead:
Library publishing takes approximately 20 minutes, as the environment needs to be built and distributed across the cluster. Session startup can take around 5 minutes, especially if the environment includes multiple or large dependencies.
To manage this efficiently, we use hybrid approach:
If this post helps, then please consider to Accept as the solution to help the other members find it more quickly and a kudos would be appreciated.
Thank you.
Hi @smpa01 ,
May I ask if you have resolved this issue? If so, please mark the helpful reply and accept it as the solution. This will be helpful for other community members who have similar problems to solve it faster.
Thank you.
Hi @smpa01 ,
Thank you for reaching out Microsoft fabric community forum and providing detailed context, including links and screenshots it’s very helpful.
You are correct in observing that libraries installed via %pip install is only available within the current notebook session. In Microsoft Fabric, %pip performs a session-scoped installation, which means it will not persist or carry over to other notebooks or Spark jobs, even within the same workspace.
Libraries like requests or aiohttp may appear persistent because they are pre-installed in Fabric’s base Spark runtime.
To achieve your goal of using termcolor (or any other third-party package) across multiple notebooks consistently, we recommend installing the package as a workspace library. Here’s how:
Once attached, it will be accessible to all notebooks and Spark jobs within that workspace.
Hope this helps. Please reach out for further assistance.
If this post helps, then please consider to Accept as the solution to help the other members find it more quickly and a kudos would be appreciated.
Thank you.
Can you give me a screnshot of this part
3. Attach it via Workspace > Libraries > Add.
Hi @smpa01 ,
Here is the screenshot showing how to attach a .whl file via:
Workspace > Libraries > Add > Custom Library
As shown above, after uploading the library, make sure to click “Publish” (top right) to finalize the changes. Only then will the custom library become available across all notebooks in the workspace.
Thank you.
The exact screen is not available to me cause Workspace > Libraries > Add > Custom Library is blocked by IT.
However, I can do probaly the same thing by creating a custom environment and by uploading the .whls there. I can attach the custom env to all my notebooks.
However,
do I understand
that I still need to run `%pip install termcolor` on the top cell of the notebook(NB1) (I think that is the case)
or
if it is once installed, it is automatically available in other notebooks (simply `import termcolor`) as well as any other subsequest execution of NB1 without needing `%pip install termcolor` in the top cell (desired).
Hello @smpa01 ,
Thanks for the clarification, you're absolutely on the right track with your approach.
When you create a custom environment and add the termcolor package by uploading its .whl file during setup, the package becomes a built-in part of that environment. This means you don’t need to run %pip install termcolor every time just attaching the environment to your notebook is enough.
You can simply use import termcolor and it will work right away. This applies to any notebook using the same environment, now or in the future, making it a consistent and hassle-free setup. If your access to the Library UI is blocked by IT, using a custom environment is a reliable and practical way to manage shared Python packages across notebooks.
If this post helps, then please consider to Accept as the solution to help the other members find it more quickly and a kudos would be appreciated.
Thank you.
When you create a custom environment and add the termcolor package by uploading its .whl file during setup, the package becomes a built-in part of that environment. This means you don’t need to run %pip install termcolor every time just attaching the environment to your notebook is enough.
You can simply use import termcolor and it will work right away. This applies to any notebook using the same environment, now or in the future, making it a consistent and hassle-free setup.
- this has severe performance implication. I created a cust env with aiofiles and termcolor. Also, The session starts in 5 minutes apart from library publishing takes ~20min (so each cust updates will be ~20 min). Can you please conf?
I am afraid to add more and have a comprehensive cust ENV for all PROD notebooks. If that is the case, I will have to resort to some other option.
Hi @smpa01 ,
You're currect, using a custom environment does introduce performance overhead:
Library publishing takes approximately 20 minutes, as the environment needs to be built and distributed across the cluster. Session startup can take around 5 minutes, especially if the environment includes multiple or large dependencies.
To manage this efficiently, we use hybrid approach:
If this post helps, then please consider to Accept as the solution to help the other members find it more quickly and a kudos would be appreciated.
Thank you.
Hi @smpa01 ,
I wanted to check if you had the opportunity to review the information provided. Please feel free to contact us if you have any further questions. If my response has addressed your query, please accept it as a solution and give a 'Kudos' so other members can easily find it.
Thank you.
Hi @smpa01 ,
I wanted to check if you had the opportunity to review the information provided. Please feel free to contact us if you have any further questions. If my response has addressed your query, please accept it as a solution and give a 'Kudos' so other members can easily find it.
Thank you.
User | Count |
---|---|
6 | |
4 | |
3 | |
2 | |
2 |
User | Count |
---|---|
17 | |
17 | |
9 | |
6 | |
6 |