Forum Discussion

Anonymous's avatar
Anonymous
Not applicable
2 years ago

Slow starter pool session initialization

I am currently experimenting with a F2 capacity. Whenever I try to run a notebook it takes somewhere between 2 and 2.5 minutes to start the spark cluster even though I'm using the starter pool. All s...
admin
data engineering
Anonymous's avatar
Anonymous
Not applicable
2 years ago

Hi Anonymous,

Apologies for the issue you have been facing.
We are reaching out to the internal team to get more information related to your query and will get back to you as soon as we have an update.

Anonymous's avatar
Anonymous
Not applicable
2 years ago

Hi Anonymous ,

Apologies for the issue you have been facing. We have an update from the team.
 We got impacted by this limitation on supporting 1 node starter pools mainly for F2 capacity sku. Team have added this to Known Issues section in the document and the latest version should be published end of this week. We are working on a fix to enable 5 second session start experience for F2 capacity. I will update this thread once the fix is rolled out in few weeks.

Thank you

  • arjaano's avatar
    arjaano
    Advocate II
    2 years ago

    This should not be an accepted solution. The issue is still there. I'm on a F4 and facing 2+ minutes startup time.

    • wiegelman's avatar
      wiegelman
      Frequent Visitor
      2 years ago

      Agree, this should not be closed - we are also using F4 and the Spark Session is not starting, it is actually timing out for us...

      Diagnostics below:

      Diagnostic ID: 22a1d13c-d334-4828-8215-dfe8e65dfb60

      Timestamp: 2024-02-28T13:38:08.717Z

      Message: [object CloseEvent]

      JSON
      {
      "type": "close",
      "timeStamp": 731861.099999994,
      "code": 1000,
      "reason": "{\"reason\":\"Session error or stopped.\",\"state\":\"session-completed\"}",
      "wasClean": true,
      "target": {
      "protocolsProfile": [
      7,
      4173
      ]
      },
      "currentTarget": {
      "protocolsProfile": [
      7,
      4173
      ]
      },
      "isTrusted": true
      }

      Additional info: InstanceId: 2491b595-9a6b-4066-bd0d-0e41152e1754

      • alozovoy's avatar
        alozovoy
        Advocate II
        2 years ago

        This is a separate issue than the rest of in this thread. If you search the forums for "Livy session has failed" you will see a number of discussions.

  • cheresier's avatar
    cheresier
    Frequent Visitor
    2 years ago

    Anonymous , has the fix been implemented? It has been ~8 weeks since you said that the fix would be "rolled out in few weeks" and I am still seeing slow startup times on the F2 capacity.

     

    Thanks

    • Anonymous's avatar
      Anonymous
      Not applicable
      2 years ago

      Hi cheresier ,

      We are reaching out to the internal team to get more information related to your query and will get back to you as soon as we have an update.

      • Anonymous's avatar
        Anonymous
        Not applicable
        2 years ago

        Now there's been another two weeks. Any update on this? Many thanks!

  • ramonsuarez's avatar
    ramonsuarez
    Advocate II
    2 years ago

    I'm getting even slower times to be able to run a notebook, up to over 8 minutes. On average they start in about 2'45", but now I'm getting more and more startups that take much longer. 

     

    Changing to an F4 capacity did not change anything. 

     

    This issue is not solved and should be changed so that it reflects that it is still open. 

  • cailen_cg's avatar
    cailen_cg
    Frequent Visitor
    2 years ago

    I'm also experiencing extremely slow start times (2-5 minutes) when using a custom environment. My environment uses all default spark settings; all it does is install a pip package.