topic Re: Keep-Alive setting for cluster? in Data Engineering

Keep-Alive setting for cluster?

_TJ — Fri, 29 Mar 2024 13:10:34 GMT

Is there some sort of 'keep-alive' settings for Fabric clusters? Like in Databricks, where you can set a timer to stop the cluster after xx amount of idle time.

We're just starting with Fabric, and it's very annoying; after every notebook fired from a pipeline, the cluster shuts down. Since starting a cluster takes a lot of time (2-3 min), our pipelines are very slow.

Any suggestions?

Re: Keep-Alive setting for cluster?

Anonymous — Mon, 01 Apr 2024 21:45:26 GMT

Hi @_TJ ,

Thanks for using Fabric Community.
Apologies for the issue you are facing, this looks like a known issue.

Docs to refer - Configure and manage starter pools in Fabric Spark. - Microsoft Fabric | Microsoft Learn

Support for High concurrency in Pipeline should address this. This is currently planned for release around Q2 of calendar year 2024 - What's new and planned for Synapse Data Engineering in Microsoft Fabric - Microsoft Fabric | Microsoft Learn

Hope this is helpful. Please let me know incase of further queries.

Re: Keep-Alive setting for cluster?

Anonymous — Tue, 02 Apr 2024 09:22:52 GMT

Hi @_TJ ,

Hello @daniele_tiles ,

We haven’t heard from you on the last response and was just checking back to see if you query was answered.
Otherwise, will respond back with the more details and we will try to help .

Re: Keep-Alive setting for cluster?

_TJ — Wed, 03 Apr 2024 07:16:33 GMT

Sorry to hear that there is no solution yet. Glad to head that the solution is on the roadmap. Thanks for the answer!

Re: Keep-Alive setting for cluster?

Anonymous — Wed, 03 Apr 2024 07:32:35 GMT

Hi @_TJ ,

Glad to know that you got some insights on your query. Please continue using Fabric Community on your further queries.

Re: Keep-Alive setting for cluster?

JB81 — Tue, 16 Jul 2024 12:55:32 GMT

But is there any control for how long the spark cluster will stay up after the last notebook finishes?

thanks