mlongtin1
Microsoft Employee
Microsoft EmployeeStatus:
Planned
Spark startup time with Python packages install is too slow
Creating an environment gives you a nice UI to set a list of Python packages to install by default The user thinks : “Oh cool, this will be faster than doing %pip install every time I start a n...
Databricks's newish serverless compute feature seems to directly compete with Fabric's temporary differentiator, which were the fast compute session launches. Now, Fabric is again behind Databricks in terms of offering clean options for maintaining a pool of properly configured compute. At least there when you reserve compute, you get pools that are configured correctly. This is embarrassing, Microsoft!
If you are launching pipelines and need python dependencies, get ready for seriously slow compute. :/
