Forum Discussion
Spark Session Charges on allocated resources
Hi
I wanted to confirm whether the allocated resources contribute to CU consumption even when no cores are actively running. I’ve attached a screenshot for reference.
Since we are using High Concurrency on a custom pool and have jobs running every 30 minutes, we’re choosing HC to avoid startup delays. However, we also want to ensure that we’re not unnecessarily consuming CUs when the environment is idle.
Additionally, I’d like to know whether we need to explicitly close the session in the notebook at the end of the process. Since we are not manually starting the session—because it is automatically created when the environment starts—we want to understand if closing it is necessary.
Thank you
Hi nagan277 , thanks for you this very details description of your issue, so, there's a couple of "asks", so, with regards to ...
#1. Do allocated resources consume CUs when idle in High Concurrency mode? - Answer is YES, when you enable High Concurrency, the environment stays always-on to avoid startup delays, this means the allocated resources continue to consume Capacity Units even if no cores are actively running jobs... in other words, you’re paying for the convenience of zero startup time. So, basically you need accept either the extra cost for High Concurrency or the startup overhead for each job 😅 (if know, tought one!)
#2. Do you need to explicitly close the notebook session? - It would be STRONGLY RECOMMENDED for you to do so if you want to save some CU's! The entire lifespan of the Spark session (from start until closed/times out) counts towards CU consumption. If you leave the session open after your job finishes, you’ll continue to incur CU charges until the session times out, so one additional best practices is to reduce the timeout setting for your sessions.
Well there you go 😊 ... hope you find this information usefull and I would appreciate a kudos for the information and even better, mark this as a solution to your question! All the very best mate, good luck!
4 Replies
- svenchio
Super User
Hi nagan277 , thanks for you this very details description of your issue, so, there's a couple of "asks", so, with regards to ...
#1. Do allocated resources consume CUs when idle in High Concurrency mode? - Answer is YES, when you enable High Concurrency, the environment stays always-on to avoid startup delays, this means the allocated resources continue to consume Capacity Units even if no cores are actively running jobs... in other words, you’re paying for the convenience of zero startup time. So, basically you need accept either the extra cost for High Concurrency or the startup overhead for each job 😅 (if know, tought one!)
#2. Do you need to explicitly close the notebook session? - It would be STRONGLY RECOMMENDED for you to do so if you want to save some CU's! The entire lifespan of the Spark session (from start until closed/times out) counts towards CU consumption. If you leave the session open after your job finishes, you’ll continue to incur CU charges until the session times out, so one additional best practices is to reduce the timeout setting for your sessions.
Well there you go 😊 ... hope you find this information usefull and I would appreciate a kudos for the information and even better, mark this as a solution to your question! All the very best mate, good luck!
Thank you svenchio lbendlin v-dineshya much detailed information. If i remember correctly, when i spoke to Microsoft Support team on this they said idle session won't cost us(It was few months ago). We needed this High Concurrency because of the near real-time data processing and i have one custom library which is used to publish the logs to App Insights. If i close the session in notebook, the HC session will be killed and it will take 10 to 13 mins to start. So i am not initiating the close.
One other issue we have here is, In many cases Graph is not showing whether dynamic allocation is working. Graph always flat it could be 40 or 50 cores being used, if i refreshed after few seconds it shows as 20 cores even for past dates. If dynamic allocation works well, It won't cost us much i think but it is not happening it seems
Thanks again
- lbendlin
Since we are using High Concurrency on a custom pool and have jobs running every 30 minutes, we’re choosing HC to avoid startup delays. However, we also want to ensure that we’re not unnecessarily consuming CUs when the environment is idle.These things are mutually exclusive. Either you save money and accept startup delays, or you pay through the nose and have an always-on environment. There is no other choice.
v-dineshyaCommunity Support
Hi nagan277 ,
Thank you for reaching out to the Microsoft Community Forum.
Hi svenchio and lbendlin , thank you for your prompt responses.
Hi nagan277 , could you please try the proposed solutions shared by lbendlin and svenchio ? Let us know if you’re still facing the same issue we’ll be happy to assist you further.
Regards,
Dinesh
