topic Re: Update Built-in libraries in new Environment in Data Science

Update Built-in libraries in new Environment

Anonymous — Fri, 14 Mar 2025 09:04:35 GMT

Hi everyone,

I am working in notebooks in Fabric, and I want to update a built-in library to a newer version, by creating a new environment and installing the newer version through PyPI.

However, when I start a session with the new notebook, it still uses the same version as in the public library (transformers 4.37.2 instead of 4.49.0).

I have also tried installing it through %pip and !pip install. This works in the current session but does not install when I create a new version. I have been given admin rights, but that does not help either. Has anyone experienced this?

Re: Update Built-in libraries in new Environment

nilendraFabric — Fri, 14 Mar 2025 14:28:37 GMT

Hello Anonymous

Few things to check :

1) Please make sure in new env , publish of library is successful (Not just saved)

2) selectthe environment from the Environment dropdown in your notebook

Re: Update Built-in libraries in new Environment

Anonymous — Mon, 17 Mar 2025 07:18:07 GMT

Hi,

I have done both, but unfortunately, that it is not the issue.

Re: Update Built-in libraries in new Environment

Anonymous — Tue, 18 Mar 2025 09:34:28 GMT

Hi Anonymous,

Thank you for reaching out in Microsoft Community Forum.

Thank you nilendraFabric for the helpful response.

The reason you’re seeing the older library version after restarting the session is because Fabric notebooks use session-based environments, which do not persist custom library installations across sessions.

Please follow below steps to resolve the error;

1. please make sure the updated version is available, run the following at the start of your notebook:
%pip install transformers==4.49.0

2.Create a new environment with the updated library version by specifying the following in requirements.txt:
transformers==4.49.0

3.If you’re still seeing the older version, clear the session cache by restarting the kernel or using:
!pip cache purge

Please continue using Microsoft community forum.

If you found this post helpful, please consider marking it as "Accept as Solution" and give it a 'Kudos'. if it was helpful. help other members find it more easily.

Regards,
Pavan.

Re: Update Built-in libraries in new Environment

Anonymous — Tue, 18 Mar 2025 12:05:09 GMT

Hi,

I have followed step number 2 the entire time, but whenever I choose the new environment, I still get the old package versions. The command !pip cache purge does not resolve the issue. Best regards.

Re: Update Built-in libraries in new Environment

Anonymous — Wed, 19 Mar 2025 05:13:19 GMT

Hi Anonymous,

Thank you for reaching out in Microsoft Community Forum.

Please follow below steps to resolve the error;

1. Run the following command to confirm the notebook is using the correct environment, If it still points to the base environment, reattach the new environment.
!which python

2. Close the notebook completely > Reopen it, go to Kernel > Change Environment, select the new environment, and restart the notebook.

3.Check if the correct version is loaded by running:
import transformers
print(transformers.__version__)

If the version is still incorrect, force reinstall it:
!pip install --force-reinstall transformers==4.49.0

Re: Update Built-in libraries in new Environment

Anonymous — Wed, 19 Mar 2025 08:44:41 GMT

Hi,

When I run the command `!which python` in my own environment, I consistently get the following output:
`/home/trusted-service-user/cluster-env/trident_env/bin/python`
This happens regardless of whether I use my own environment or the default one. I have reattached my own environment multiple times, closed the notebook, restarted the session, etc., but with no changes. It seems I am unable to attach my own environment to the notebook. I have also tried to recreate the environment, but I encountered the same result.

Re: Update Built-in libraries in new Environment

Anonymous — Wed, 19 Mar 2025 09:43:58 GMT

Hi Anonymous,

Thank you for the detailed update. Please follow below steps to resolve the error;

1.Even if you created the environment, it won’t be applied unless you publish it.
Go to Fabric > Environments → Verify your environment is "Published" (not just "Saved").

2.Manually Attach the Environment, Please run the below command
!source /home/trusted-service-user/cluster-env/<your_env_name>/bin/activate

3.Force Reinstall the Library and Verify:
For installation; %pip install --force-reinstall transformers==4.49.0

For Verification;
import transformers
print(transformers.__version__)

Re: Update Built-in libraries in new Environment

Anonymous — Wed, 19 Mar 2025 10:14:51 GMT

Hi,

The packages are all published:

But I still cant connect to the environment:

Re: Update Built-in libraries in new Environment

Anonymous — Wed, 19 Mar 2025 11:38:48 GMT

Hi Anonymous,

Thank you for the update and sharing the screenshots.

1. Please check the fabric permissions
Go to Admin Portal > Tenant Settings > Environments → Ensure you have "Create and Manage Environments" permissions.

2.Manually Set Environment in Notebook Metadata
Go to Notebook Settings > Advanced > Metadata → Add

{
"kernel": {
"name": "python3",
"environment": "<your_env_name>"
}
}

3.Create a new notebook, select your environment, and verify with below code
!which python

Please continue using Microsoft community forum.

If you found this post helpful, please consider marking it as "Accept as Solution" and give it a 'Kudos'. if it was helpful. help other members find it more easily.

Regards,
Pavan.

Re: Update Built-in libraries in new Environment

Anonymous — Wed, 19 Mar 2025 13:04:06 GMT

Hi,

We do not see Environments in Tenant Settings, or Advanced in Notebook Settings:

Re: Update Built-in libraries in new Environment

Anonymous — Fri, 21 Mar 2025 15:42:56 GMT

Hi Anonymous,

Thank you for reaching out in Microsoft Community Forum.

please follow below steps to resolve the issue;

1. Since you can't see Environments in Tenant Settings, it indicates you lack admin privileges. Ask your Fabric Admin to grant you access to "Create and Manage Environments" in Tenant Settings.

2. Make sure you are using a Premium workspace with Fabric capacity enabled.

3. Manually activate the environment in the current session using:
!source /home/trusted-service-user/cluster-env/<your_env_name>/bin/activate
!pip install --force-reinstall transformers==4.49.0

Re: Update Built-in libraries in new Environment

Anonymous — Tue, 25 Mar 2025 01:12:57 GMT

Hi Anonymous,

I wanted to check if you had the opportunity to review the information provided. Please feel free to contact us if you have any further questions. If my response has addressed your query, please "Accept as Solution" and give a 'Kudos' so other members can easily find it.

Thank you,
Pavan.

Re: Update Built-in libraries in new Environment

Anonymous — Tue, 25 Mar 2025 07:52:24 GMT

Hi,

1. The pictures I sent in my previous reply are from our admin user, so the admins cannot see the settings you mentioned.
2. I do have a Fabric capacity.

Best regards

Re: Update Built-in libraries in new Environment

Anonymous — Tue, 25 Mar 2025 12:12:08 GMT

Hi Anonymous,

Thank you for reaching out in Microsoft Community Forum.

Please go through the above mentioned steps and check step by step again ,if still faces the same issue then please consider raising a support ticket for further assistance.

To raise a support ticket for Fabric and Power BI, kindly follow the steps outlined in the following guide:
https://learn.microsoft.com/en-us/power-bi/support/create-support-ticket

Re: Update Built-in libraries in new Environment

Pradeep_O_S — Tue, 08 Apr 2025 03:26:03 GMT

Unfortunately, there is currently no way to override the runtime packages in Fabric. Only session-level libraries can be modified using %pip and !pip install.

Re: Update Built-in libraries in new Environment

Anonymous — Tue, 08 Apr 2025 13:50:09 GMT

So the Library Management only work for packages that are not in built-in?

Re: Update Built-in libraries in new Environment

Pradeep_O_S — Thu, 10 Apr 2025 00:08:42 GMT

It seems so. I encountered the same issue you did. Installing a newer version at the environment level doesn’t appear to have any effect. My hypothesis is that Microsoft is intentionally restricting this to avoid potential dependency conflicts with the standard packages bundled in the environment.

Re: Update Built-in libraries in new Environment

mkj1213 — Thu, 19 Jun 2025 09:50:50 GMT

I am having exactly the same issue as described in the original post,

i had created (saved and published) a environment with the following custom libraries:

however when i am running:

I still see that i am using the built in libraries:

any one had any luck solving this problem?

Re: Update Built-in libraries in new Environment

big_idea — Mon, 23 Jun 2025 22:33:33 GMT

I have experienced this. It's baffling that nobody suggested this, but unfortunately this page reads like 99.9 of all requests for help with Microsoft I've ever seen. Conda packages override pypy ones. Try adding the packages needed from conda instead of pypy in the environment package area. You should see the option at the top left in the same screen that you posted the screenshot of.