Forum Discussion

abhidotnet's avatar
abhidotnet
Advocate II
7 months ago

Faker Works in PySpark… but Not Python

I ran into while trying to use the Faker library inside both PySpark and Python notebooks in Microsoft Fabric. I’m hoping this helps others, and I’d love confirmation from the product team on whether...
workspaces
v-echaithra's avatar
v-echaithra
Community Support
6 months ago

Hi abhidotnet ,

Thank you for reaching out to Microsoft Community.
At the moment, this is a current limitation of Microsoft Fabric rather than a configuration issue. Python notebooks don’t yet support attaching custom Environments, while PySpark notebooks do.
Right now, you must use %pip install in Python notebooks or use PySpark notebooks to attach environments.

Since this affects dependency management and consistency across notebook types, I’d recommend raising a feature request on Fabric Ideas so the product team can track demand and prioritize Python-runtime support for Environments. It’s likely the best way to get visibility and an official response on roadmap timing.

You’re welcome to post this in the Ideas forum here: Fabric Ideas - Microsoft Fabric Community

That’s where enhancement suggestions go. The Power BI team actively reviews and prioritizes ideas based on community feedback and votes.

Thank you.

    • v-echaithra's avatar
      v-echaithra
      Community Support
      6 months ago

      Hi abhidotnet ,

      Thank you for submitting this as a feature request and sharing the link. The product team will review and evaluate this. We appreciate you taking the time to help improve the Fabric experience.

      Thanks again for your contribution

  • mrbartuss's avatar
    mrbartuss
    Advocate II
    5 months ago

    deborshi_nag v-echaithra 
    One follow-up for production use - if this pure Python notebook is scheduled in a Data Factory pipeline, is inline %pip install completely safe? I just want to make sure automated runs won't hit any of the stability issues mentioned here: https://learn.microsoft.com/en-us/fabric/data-engineering/library-management#python-inline-installat... warning against inline pip (even though I know those articles mostly focus on Spark)

    • deborshi_nag's avatar
      deborshi_nag
      Super User
      5 months ago

      Hello mrbartuss %pip command restarts the Python interpreter, as long as you keep that statement as the first line of code in your notebook, it should be fine for production workloads using data pipelines.

       

      I would also recommend that you specify a specific version (or a range) of a Python library when using %pip. This reduces risk in your production pipelines in case a new version of the library is made available.

       

      %pip install numpy==1.26.4


      %pip install "pandas>=1.5,<2.0"

       

    • v-echaithra's avatar
      v-echaithra
      Community Support
      5 months ago

      Hi mrbartuss ,

      The warning in the documentation you referenced mainly applies to Spark notebooks, where inline installation can trigger Spark session restarts and impact long-running jobs. For pure Python notebooks, %pip install is currently the supported approach for managing dependencies while the Python runtime experience is still in preview.

      Hope this helps.
      Chaithra E.