mlongtin1
Microsoft Employee
Microsoft EmployeeStatus:
Planned
Spark startup time with Python packages install is too slow
Creating an environment gives you a nice UI to set a list of Python packages to install by default The user thinks : “Oh cool, this will be faster than doing %pip install every time I start a n...
forget 3 minutes - if you have custom network (because for some reason your production is not available to the entire internet), and you have a library you can expect 10 to 15 minutes start-up, which support says is "expected" it is currently documented at 'https://learn.microsoft.com/en-us/fabric/data-engineering/spark-compute' that you can expect 2 to 5 minutes extra for adding a library, but i find that i pay the full 5 minutes even for a little helper library that i can pip install in less than 30 seconds.
