Forum Discussion
Advocate IIIs %pip install safe for a Python notebook used in a pipeline?
Hi MR
Yes, Python notebooks don't support environments yet. And inline %pip install is the officially recommended best practice now.
Please see the documentation:
You can use %pip and %conda commands for inline installations, the commands support both public libraries and customized libraries.
For customized libraries, you can upload the lib files to the Built-in resources folder. We support multiple types of libraries, including formats such as Wheel (.whl), JAR (.jar), DLL (.dll), and Python (.py). Just try drag&drop to the file and the code snippet is generated automatically.
The link to it:
And one more point regarding driver/worker. Python Notebook's architecture is much simplier than Spark notebook, it is their pro. They don't have driver/worker, that's why they use 2 VCores per time comparing to 8+ for Spark notebooks. So, the issue with spreading installation accross drivers/workers should not be an issue.
I hope this helped! If it answered your question, please mark it as a Solution so others with the same issue can find it easily. Kudos are also always appreciated!
BR, Yurri
Advocate IIFor PySpark workloads in Fabric, custom environments are the recommended standard over inline %pip install to avoid driver/worker sync issues (source). Pure Python notebooks currently lack custom environment support. Is inline %pip install the officially recommended best practice for installing libraries like semantic-link-labs? Thanks for the clarification!
v-hashadapuCommunity Support
Hi mrbartuss , Thank you for reaching out to the Microsoft Community Forum.
For now, yes. Since Python notebooks in Fabric don’t yet support custom environments, installing libraries inline, for example with %pip install semantic-link-labs is currently the supported approach for adding dependencies in those notebooks. The recommendation to use custom environments applies to Spark/PySpark workloads, where environments help manage dependencies across the distributed driver and executors.
Library Management in Fabric Environments - Microsoft Fabric | Microsoft Learn
Manage Apache Spark libraries - Microsoft Fabric | Microsoft Learn