Forum Discussion
Faker Works in PySpark… but Not Python
deborshi_nag v-echaithra
One follow-up for production use - if this pure Python notebook is scheduled in a Data Factory pipeline, is inline %pip install completely safe? I just want to make sure automated runs won't hit any of the stability issues mentioned here: https://learn.microsoft.com/en-us/fabric/data-engineering/library-management#python-inline-installat... warning against inline pip (even though I know those articles mostly focus on Spark)
Hello mrbartuss %pip command restarts the Python interpreter, as long as you keep that statement as the first line of code in your notebook, it should be fine for production workloads using data pipelines.
I would also recommend that you specify a specific version (or a range) of a Python library when using %pip. This reduces risk in your production pipelines in case a new version of the library is made available.
%pip install numpy==1.26.4
%pip install "pandas>=1.5,<2.0"