Spark startup time with Python packages install is too slow

We had the same problem. The moment we customize the node with PyPi packages, we loose the StarterPools instant start feature and go back to old 3-4 minute start time.

For sometime, used the pip to install the packages from inside the notebook.

And one file day, all ETLs failed stating the %pip magic commands disabled inside pipeline.

Finally i managed to install the pip packages into Lakehouse files space like below

!pip install googleads -t /lakehouse/default/Files/PyPi Packages/

and then in the notebook i add below line to include the Lakehouse folder into syspath so the package can be imported like local installed package.

import sys

sys.path.append('/lakehouse/default/Files/PyPi Packages/')

This is a workaround to avoid high start time and still work with custom packages.

Hope there will be some native fix for this in the future fabric runtimes.