Forum Discussion

benitovbreugel's avatar
benitovbreugel
Icon for Advocate II rankAdvocate II
11 months ago

Issue spark resource usage is low & inefficient

Hi,    We are running on an f8 Fabric capacity, with our own created spark environment. In our pipeline we are running the same notebook multiple times and it always uses the created environment.  ...
data engineering
environment
benitovbreugel's avatar
benitovbreugel
Icon for Advocate II rankAdvocate II
10 months ago

Hi, 

 

At the moment we have increased efficiency from 17% tot 33%. However, still encoutering an additional 10 minutes before the session actually does something. 

 

We have done the following: 

  • Confirm if you’re queued or just slow to start. It seems
  • Minimize custom environment cost. We changed some notebooks to the standard environments, but not all according to some installation packages we need.
  • Reuse one session. Which is someting we do with High Concurrency
  • Reduce “many notebooks” > “one session”.  This is also what we try to enforce. However It seems that everytime a new session is started, which should not be the case.

So in total we are getting better, but are not there yet.

 

 

v-prasare's avatar
v-prasare
Icon for Community Support rankCommunity Support
10 months ago

Hi benitovbreugel,

Thanks for the your valuable insights, these will be useful to Fabric community users with similar issues.

 

If your getting blocked here, please consider raising a support ticket for further assistance. To raise a support ticket for Fabric and Power BI, kindly follow the steps outlined in the following guide:

How to create a Fabric and Power BI Support ticket - Power BI | Microsoft Learn

 

 

 

Thanks,

Prashanth