Forum Discussion

benitovbreugel's avatar
benitovbreugel
Icon for Advocate II rankAdvocate II
11 months ago

Issue spark resource usage is low & inefficient

Hi,    We are running on an f8 Fabric capacity, with our own created spark environment. In our pipeline we are running the same notebook multiple times and it always uses the created environment.  ...
data engineering
environment
v-prasare's avatar
v-prasare
Icon for Community Support rankCommunity Support
10 months ago

Hi benitovbreugel,

We would like to confirm if our community members answer resolves your query or if you need further help. If you still have any questions or need more support, please feel free to let us know. We are happy to help you.

 

 

Thank you for your patience and look forward to hearing from you.
Best Regards,
Prashanth Are
MS Fabric community support

benitovbreugel's avatar
benitovbreugel
Icon for Advocate II rankAdvocate II
10 months ago

Hi, 

 

At the moment we have increased efficiency from 17% tot 33%. However, still encoutering an additional 10 minutes before the session actually does something. 

 

We have done the following: 

  • Confirm if you’re queued or just slow to start. It seems
  • Minimize custom environment cost. We changed some notebooks to the standard environments, but not all according to some installation packages we need.
  • Reuse one session. Which is someting we do with High Concurrency
  • Reduce “many notebooks” > “one session”.  This is also what we try to enforce. However It seems that everytime a new session is started, which should not be the case.

So in total we are getting better, but are not there yet.