mlongtin1's avatar
mlongtin1
Icon for Microsoft Employee rankMicrosoft Employee
2 years ago
Status:
Planned

Spark startup time with Python packages install is too slow

Creating an environment gives you a nice UI to set a list of Python packages to install by default The user thinks : “Oh cool, this will be faster than doing %pip install every time I start a n...
data engineering
Andrew_Hill_TRS's avatar
Andrew_Hill_TRS
Regular Visitor
11 months ago
Anything more that 15 minutes is out of spec, and that would by for a custom size, vnet linked environment with a custom library added, each documented as *max* 5 minutes, but minimum 30b seconds to 2 minutes.

Recent ideas