Spark startup time with Python packages install is too slow

We had the same problem. The moment we customize the node with PyPi packages, we loose the StarterPools instant start feature and go back to old 3-4 minute start time.

For sometime, used the pip to install the packages from inside the notebook.

And one file day, all ETLs failed stating the %pip magic commands disabled inside pipeline.

Finally i managed to install the pip packages into Lakehouse files space like below

!pip install googleads -t /lakehouse/default/Files/PyPi Packages/

and then in the notebook i add below line to include the Lakehouse folder into syspath so the package can be imported like local installed package.

import sys

sys.path.append('/lakehouse/default/Files/PyPi Packages/')

This is a workaround to avoid high start time and still work with custom packages.

Hope there will be some native fix for this in the future fabric runtimes.

Databricks's newish serverless compute feature seems to directly compete with Fabric's temporary differentiator, which were the fast compute session launches. Now, Fabric is again behind Databricks in terms of offering clean options for maintaining a pool of properly configured compute. At least there when you reserve compute, you get pools that are configured correctly. This is embarrassing, Microsoft!

If you are launching pipelines and need python dependencies, get ready for seriously slow compute. :/