Forum Discussion
library installation issue pyenv vs cluster-env
Hi smpa01 ,
You're currect, using a custom environment does introduce performance overhead:
Library publishing takes approximately 20 minutes, as the environment needs to be built and distributed across the cluster. Session startup can take around 5 minutes, especially if the environment includes multiple or large dependencies.
To manage this efficiently, we use hybrid approach:
- Use the custom environment for stable, shared libraries that are unlikely to change frequently (e.g., termcolor, numpy, pandas). This ensures consistency across notebooks without frequent rebuilds.
- Install fast-changing or notebook-specific libraries (like aiofiles) using %pip install in the first cell of the notebook. This avoids the need to republish the entire environment for every update.
If this post helps, then please consider to Accept as the solution to help the other members find it more quickly and a kudos would be appreciated.
Thank you.
When you create a custom environment and add the termcolor package by uploading its .whl file during setup, the package becomes a built-in part of that environment. This means you don’t need to run %pip install termcolor every time just attaching the environment to your notebook is enough.
You can simply use import termcolor and it will work right away. This applies to any notebook using the same environment, now or in the future, making it a consistent and hassle-free setup.
- this has severe performance implication. I created a cust env with aiofiles and termcolor. Also, The session starts in 5 minutes apart from library publishing takes ~20min (so each cust updates will be ~20 min). Can you please conf?
I am afraid to add more and have a comprehensive cust ENV for all PROD notebooks. If that is the case, I will have to resort to some other option.
Hi smpa01 ,
You're currect, using a custom environment does introduce performance overhead:
Library publishing takes approximately 20 minutes, as the environment needs to be built and distributed across the cluster. Session startup can take around 5 minutes, especially if the environment includes multiple or large dependencies.
To manage this efficiently, we use hybrid approach:
- Use the custom environment for stable, shared libraries that are unlikely to change frequently (e.g., termcolor, numpy, pandas). This ensures consistency across notebooks without frequent rebuilds.
- Install fast-changing or notebook-specific libraries (like aiofiles) using %pip install in the first cell of the notebook. This avoids the need to republish the entire environment for every update.
If this post helps, then please consider to Accept as the solution to help the other members find it more quickly and a kudos would be appreciated.
Thank you.
Hi smpa01 ,
I wanted to check if you had the opportunity to review the information provided. Please feel free to contact us if you have any further questions. If my response has addressed your query, please accept it as a solution and give a 'Kudos' so other members can easily find it.
Thank you.Hi @smpa01 ,
I wanted to check if you had the opportunity to review the information provided. Please feel free to contact us if you have any further questions. If my response has addressed your query, please accept it as a solution and give a 'Kudos' so other members can easily find it.
Thank you.