mlongtin1's avatar
mlongtin1
Icon for Microsoft Employee rankMicrosoft Employee
2 years ago
Status:
Planned

Spark startup time with Python packages install is too slow

Creating an environment gives you a nice UI to set a list of Python packages to install by default The user thinks : “Oh cool, this will be faster than doing %pip install every time I start a n...
data engineering
nielsvdc's avatar
nielsvdc
Icon for Super User rankSuper User
11 months ago

Currently in our environment startup time for a custom environment on a starter pool can take up to 18 minutes. This is since the September release after FabCon Europe. 18 minutes is a lot of coffee drinking and not really being productive. I'm getting questions about when we are moving to Synapse or Databricks, which is not what we really want from an architecture point of view.

Update: There is a known issue since August https://support.fabric.microsoft.com/known-issues/?active=true&fixed=true&sort=published&product=Data%2520Engineering&issueId=1550

Recent ideas