Issue spark resource usage is low & inefficient

Hi tayloramy,

Thanks for your clarification. 

We are indeed using a custom environments, so trying to minimize the usage of this in some notebooks. 

Next to that we also try to reuse the same session with high concurrency. 

The last is that we now also challenge the capacity presure. Maybe in total we are ok, but at the moment we run our process it is possible we exceeds limit at that moent. 

Will investigate this further.