Keep-Alive setting for cluster?

[_TJ]

Is there some sort of 'keep-alive' settings for Fabric clusters? Like in Databricks, where you can set a timer to stop the cluster after xx amount of idle time.

We're just starting with Fabric, and it's very annoying; after every notebook fired from a pipeline, the cluster shuts down. Since starting a cluster takes a lot of time (2-3 min), our pipelines are very slow. 

Any suggestions?

[_TJ]

Sorry to hear that there is no solution yet. Glad to head that the solution is on the roadmap. Thanks for the answer!

[Anonymous]

Hi @_TJ ,

Glad to know that you got some insights on your query. Please continue using Fabric Community on your further queries.

[Anonymous · Accepted solution]

Hi @_TJ ,

Thanks for using Fabric Community.
Apologies for the issue you are facing, this looks like a known issue.

Docs to refer - Configure and manage starter pools in Fabric Spark. - Microsoft Fabric | Microsoft Learn

Support for High concurrency in Pipeline should address this. This is currently planned for release around Q2 of calendar year 2024 - What's new and planned for Synapse Data Engineering in Microsoft Fabric - Microsoft Fabric | Microso...

Hope this is helpful. Please let me know incase of further queries.

[JB81]

But is there any control for how long the spark cluster will stay up after the last notebook finishes?

thanks

JB

[Anonymous]

Hi @_TJ ,

Hello @daniele_tiles ,

We haven’t heard from you on the last response and was just checking back to see if you query was answered.
Otherwise, will respond back with the more details and we will try to help .