Reg: Capacity consumption for fabric Notebook

[Mathan1787]

Hi,

I am recently trying out spark structured streaming in fabric notebook and try to check if its processing the files in real time. My notebook is running for past 2 days with very minimum vcore and Memory for both driver(1) and executor (1) via environment.   In my previous try, i didn't have environment and job ran for 3 days with f64 capacity and at one point capacity consumation is 100%. Capacity consumation is keep increasing for each day irrespective of new environment. if the job runs for a long time does it cost more CU. I dont really understand this. could someone help here. (is it not wise to use structured streaming)

[Ray_Minds]

Hi @Mathan1787 

Here you will do two things.

1. First approach:- If possible to you , you will the increase your capacity. F64 to another license.
   https://learn.microsoft.com/en-us/fabric/enterprise/licenses
2. Second approach:- You must be check your all jobs scheduled timing. it should be different to each other. it will reduce the load. if you have issue with particular notebook. you need to do performance tuning for the particular notebook.
   
   Please feel free to ping us if you need any help further.

If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.

[Anonymous]

Hi @Mathan1787 ,

Any update on this? Did the above suggestions help with your scenario? if that is the case, you can consider Kudo or Accept the helpful suggestions to help others who faced similar requirements.

Regards,

Xiaoxin Sheng

[Anonymous]

Hi @Mathan1787,

I'd like to suggest you check the official document about capacity throttling if helps for your scenario:

Understand your Fabric capacity throttling - Microsoft Fabric | Microsoft Learn

Regards,
Xiaoxin Sheng