Forum Discussion
Update Built-in libraries in new Environment
I have experienced this. It's baffling that nobody suggested this, but unfortunately this page reads like 99.9 of all requests for help with Microsoft I've ever seen. Conda packages override pypy ones. Try adding the packages needed from conda instead of pypy in the environment package area. You should see the option at the top left in the same screen that you posted the screenshot of.
Unfortunately, there is currently no way to override the runtime packages in Fabric. Only session-level libraries can be modified using %pip and !pip install.
I was able to solve this error using %pip in my notebook. Apparently %pip installs the required package versions across all Spark nodes - both the driver and executors - in the clonedenv environment. Unlike !pip, which only affects the driver, %pip ensures consistency in the distributed environment, fixing the transformers tokenizers version mismatch: %pip install transformers tokenizers --upgrade
Glad you solved it! Yes, %pip in notebooks installs packages across all Spark nodes in the cloned environment, ensuring consistent versions for both driver and executors—unlike !pip, which only updates the driver. This fixes version mismatches like the one with transformers and tokenizers.
