Forum Discussion
Update Built-in libraries in new Environment
I have experienced this. It's baffling that nobody suggested this, but unfortunately this page reads like 99.9 of all requests for help with Microsoft I've ever seen. Conda packages override pypy ones. Try adding the packages needed from conda instead of pypy in the environment package area. You should see the option at the top left in the same screen that you posted the screenshot of.
Unfortunately, there is currently no way to override the runtime packages in Fabric. Only session-level libraries can be modified using %pip and !pip install.
So the Library Management only work for packages that are not in built-in?
It seems so. I encountered the same issue you did. Installing a newer version at the environment level doesn’t appear to have any effect. My hypothesis is that Microsoft is intentionally restricting this to avoid potential dependency conflicts with the standard packages bundled in the environment.
I was able to solve this error using %pip in my notebook. Apparently %pip installs the required package versions across all Spark nodes - both the driver and executors - in the clonedenv environment. Unlike !pip, which only affects the driver, %pip ensures consistency in the distributed environment, fixing the transformers tokenizers version mismatch: %pip install transformers tokenizers --upgrade
Glad you solved it! Yes, %pip in notebooks installs packages across all Spark nodes in the cloned environment, ensuring consistent versions for both driver and executors—unlike !pip, which only updates the driver. This fixes version mismatches like the one with transformers and tokenizers.
