Forum Discussion
library installation issue pyenv vs cluster-env
Hi smpa01 ,
You're currect, using a custom environment does introduce performance overhead:
Library publishing takes approximately 20 minutes, as the environment needs to be built and distributed across the cluster. Session startup can take around 5 minutes, especially if the environment includes multiple or large dependencies.
To manage this efficiently, we use hybrid approach:
- Use the custom environment for stable, shared libraries that are unlikely to change frequently (e.g., termcolor, numpy, pandas). This ensures consistency across notebooks without frequent rebuilds.
- Install fast-changing or notebook-specific libraries (like aiofiles) using %pip install in the first cell of the notebook. This avoids the need to republish the entire environment for every update.
If this post helps, then please consider to Accept as the solution to help the other members find it more quickly and a kudos would be appreciated.
Thank you.
Hi smpa01 ,
May I ask if you have resolved this issue? If so, please mark the helpful reply and accept it as the solution. This will be helpful for other community members who have similar problems to solve it faster.
Thank you.