Forum Discussion

_TJ's avatar
_TJ
Frequent Visitor
2 years ago
Solved

Keep-Alive setting for cluster?

Is there some sort of 'keep-alive' settings for Fabric clusters? Like in Databricks, where you can set a timer to stop the cluster after xx amount of idle time.

 

We're just starting with Fabric, and it's very annoying; after every notebook fired from a pipeline, the cluster shuts down. Since starting a cluster takes a lot of time (2-3 min), our pipelines are very slow. 

 

Any suggestions?

data engineering

5 Replies

  • Anonymous's avatar
    Anonymous
    Not applicable
    2 years ago

    Hi _TJ ,

    Thanks for using Fabric Community.
    Apologies for the issue you are facing, this looks like a known issue.


    Docs to refer - Configure and manage starter pools in Fabric Spark. - Microsoft Fabric | Microsoft Learn

    Support for High concurrency in Pipeline should address this. This is currently planned for release around Q2 of calendar year 2024 - What's new and planned for Synapse Data Engineering in Microsoft Fabric - Microsoft Fabric | Microsoft Learn

    Hope this is helpful. Please let me know incase of further queries.

    • Anonymous's avatar
      Anonymous
      Not applicable
      2 years ago

      Hi _TJ ,

      Hello daniele_tiles ,

      We haven’t heard from you on the last response and was just checking back to see if you query was answered.
      Otherwise, will respond back with the more details and we will try to help .

    • JB81's avatar
      JB81
      Icon for Helper II rankHelper II
      2 years ago

      But is there any control for how long the spark cluster will stay up after the last notebook finishes?

       

      thanks

      JB

  • _TJ's avatar
    _TJ
    Frequent Visitor
    2 years ago

    Sorry to hear that there is no solution yet. Glad to head that the solution is on the roadmap. Thanks for the answer!

    • Anonymous's avatar
      Anonymous
      Not applicable
      2 years ago

      Hi _TJ ,

      Glad to know that you got some insights on your query. Please continue using Fabric Community on your further queries.