Forum Discussion
Update Built-in libraries in new Environment
Hi everyone,
I am working in notebooks in Fabric, and I want to update a built-in library to a newer version, by creating a new environment and installing the newer version through PyPI.
However, when I start a session with the new notebook, it still uses the same version as in the public library (transformers 4.37.2 instead of 4.49.0).
I have also tried installing it through %pip and !pip install. This works in the current session but does not install when I create a new version. I have been given admin rights, but that does not help either. Has anyone experienced this?
I have experienced this. It's baffling that nobody suggested this, but unfortunately this page reads like 99.9 of all requests for help with Microsoft I've ever seen. Conda packages override pypy ones. Try adding the packages needed from conda instead of pypy in the environment package area. You should see the option at the top left in the same screen that you posted the screenshot of.
23 Replies
Unfortunately, there is currently no way to override the runtime packages in Fabric. Only session-level libraries can be modified using %pip and !pip install.
It seems so. I encountered the same issue you did. Installing a newer version at the environment level doesn’t appear to have any effect. My hypothesis is that Microsoft is intentionally restricting this to avoid potential dependency conflicts with the standard packages bundled in the environment.
I was able to solve this error using %pip in my notebook. Apparently %pip installs the required package versions across all Spark nodes - both the driver and executors - in the clonedenv environment. Unlike !pip, which only affects the driver, %pip ensures consistency in the distributed environment, fixing the transformers tokenizers version mismatch: %pip install transformers tokenizers --upgrade
Glad you solved it! Yes, %pip in notebooks installs packages across all Spark nodes in the cloned environment, ensuring consistent versions for both driver and executors—unlike !pip, which only updates the driver. This fixes version mismatches like the one with transformers and tokenizers.
I am having exactly the same issue as described in the original post,
i had created (saved and published) a environment with the following custom libraries:
however when i am running:!pip list | grep -E "bertopic|transformers|tokenizers|huggingface|safetensors"I still see that i am using the built in libraries:
any one had any luck solving this problem?Hi Anonymous,
Thank you for reaching out in Microsoft Community Forum.
Thank you nilendraFabric for the helpful response.
The reason you’re seeing the older library version after restarting the session is because Fabric notebooks use session-based environments, which do not persist custom library installations across sessions.Please follow below steps to resolve the error;
1. please make sure the updated version is available, run the following at the start of your notebook:
%pip install transformers==4.49.02.Create a new environment with the updated library version by specifying the following in requirements.txt:
transformers==4.49.03.If you’re still seeing the older version, clear the session cache by restarting the kernel or using:
!pip cache purgePlease continue using Microsoft community forum.
If you found this post helpful, please consider marking it as "Accept as Solution" and give it a 'Kudos'. if it was helpful. help other members find it more easily.
Regards,
Pavan.Hi Anonymous,
Thank you for reaching out in Microsoft Community Forum.
Please follow below steps to resolve the error;
1. Run the following command to confirm the notebook is using the correct environment, If it still points to the base environment, reattach the new environment.
!which python2. Close the notebook completely > Reopen it, go to Kernel > Change Environment, select the new environment, and restart the notebook.
3.Check if the correct version is loaded by running:
import transformers
print(transformers.__version__)If the version is still incorrect, force reinstall it:
!pip install --force-reinstall transformers==4.49.0Please continue using Microsoft community forum.
If you found this post helpful, please consider marking it as "Accept as Solution" and give it a 'Kudos'. if it was helpful. help other members find it more easily.
Regards,
Pavan.
Hi Anonymous,
I wanted to check if you had the opportunity to review the information provided. Please feel free to contact us if you have any further questions. If my response has addressed your query, please "Accept as Solution" and give a 'Kudos' so other members can easily find it.
Thank you,
Pavan.Hello Anonymous
Few things to check :
1) Please make sure in new env , publish of library is successful (Not just saved)2) selectthe environment from the Environment dropdown in your notebook
I have experienced this. It's baffling that nobody suggested this, but unfortunately this page reads like 99.9 of all requests for help with Microsoft I've ever seen. Conda packages override pypy ones. Try adding the packages needed from conda instead of pypy in the environment package area. You should see the option at the top left in the same screen that you posted the screenshot of.
This doesn't work unfortunately because the conda that installs packages isn't conda-forge. You can only install earlier versions of transformers with conda - transformers v4.38.x. The latest stable version for transformers is currently as at this date is v4.53.2. Thus defeating the purpose. With conda-forge you can get the latest version but this is only doable within the notebook and is still at runtime so the built-in packages still cannot be overriden. What Fabric needs to do is interms of python packages is:
A. Keep their built-in environment up to date which is something they've failed to do, or
B. Allow users to be able to override their fixed built-in python packages through custom install, which is something they already do with some of their built-in packages.
None of the 20 replies before this one here worked!
Hi mkj1213 Anonymous
Thank you for being part of the Microsoft Fabric Community.
As highlighted by big_idea , the proposed approach appears to effectively address your requirements. Could you please confirm if your issue has been resolved?
If you are still facing any challenges, kindly provide further details, and we will be happy to assist you.
If the above information is helpful, please give us Kudos and mark the response as Accepted as solution.
Best Regards,
Community Support Team _ C Srikanth.
