Forum Discussion
Is %pip install safe for a Python notebook used in a pipeline?
For PySpark workloads in Fabric, custom environments are the recommended standard over inline %pip install to avoid driver/worker sync issues (source). Pure Python notebooks currently lack custom environment support. Is inline %pip install the officially recommended best practice for installing libraries like semantic-link-labs? Thanks for the clarification!
Hi MR
Yes, Python notebooks don't support environments yet. And inline %pip install is the officially recommended best practice now.
Please see the documentation:
You can use %pip and %conda commands for inline installations, the commands support both public libraries and customized libraries.
For customized libraries, you can upload the lib files to the Built-in resources folder. We support multiple types of libraries, including formats such as Wheel (.whl), JAR (.jar), DLL (.dll), and Python (.py). Just try drag&drop to the file and the code snippet is generated automatically.
The link to it:
And one more point regarding driver/worker. Python Notebook's architecture is much simplier than Spark notebook, it is their pro. They don't have driver/worker, that's why they use 2 VCores per time comparing to 8+ for Spark notebooks. So, the issue with spreading installation accross drivers/workers should not be an issue.
I hope this helped! If it answered your question, please mark it as a Solution so others with the same issue can find it easily. Kudos are also always appreciated!
BR, Yurri
6 Replies
Hi MR
Yes, Python notebooks don't support environments yet. And inline %pip install is the officially recommended best practice now.
Please see the documentation:
You can use %pip and %conda commands for inline installations, the commands support both public libraries and customized libraries.
For customized libraries, you can upload the lib files to the Built-in resources folder. We support multiple types of libraries, including formats such as Wheel (.whl), JAR (.jar), DLL (.dll), and Python (.py). Just try drag&drop to the file and the code snippet is generated automatically.
The link to it:
And one more point regarding driver/worker. Python Notebook's architecture is much simplier than Spark notebook, it is their pro. They don't have driver/worker, that's why they use 2 VCores per time comparing to 8+ for Spark notebooks. So, the issue with spreading installation accross drivers/workers should not be an issue.
I hope this helped! If it answered your question, please mark it as a Solution so others with the same issue can find it easily. Kudos are also always appreciated!
BR, Yurri
For PySpark workloads in Fabric, custom environments are the recommended standard over inline %pip install to avoid driver/worker sync issues (source). Pure Python notebooks currently lack custom environment support. Is inline %pip install the officially recommended best practice for installing libraries like semantic-link-labs? Thanks for the clarification!
Hi mrbartuss , Thank you for reaching out to the Microsoft Community Forum.
For now, yes. Since Python notebooks in Fabric don’t yet support custom environments, installing libraries inline, for example with %pip install semantic-link-labs is currently the supported approach for adding dependencies in those notebooks. The recommendation to use custom environments applies to Spark/PySpark workloads, where environments help manage dependencies across the distributed driver and executors.
Library Management in Fabric Environments - Microsoft Fabric | Microsoft Learn
Manage Apache Spark libraries - Microsoft Fabric | Microsoft Learn
Hi mrbartuss , hope you are doing great. May we know if your issue is solved or if you are still experiencing difficulties. Please share the details as it will help the community, especially others with similar issues.
Hi mrbartuss , Hope you're doing okay! May we know if it worked for you, or are you still experiencing difficulties? Let us know — your feedback can really help others in the same situation.
All clear. Thanks!
