topic Re: Reg: Capacity consumption for fabric Notebook in Data Engineering

Reg: Capacity consumption for fabric Notebook

Mathan1787 — Wed, 18 Dec 2024 18:18:04 GMT

Hi,

I am recently trying out spark structured streaming in fabric notebook and try to check if its processing the files in real time. My notebook is running for past 2 days with very minimum vcore and Memory for both driver(1) and executor (1) via environment. In my previous try, i didn't have environment and job ran for 3 days with f64 capacity and at one point capacity consumation is 100%. Capacity consumation is keep increasing for each day irrespective of new environment. if the job runs for a long time does it cost more CU. I dont really understand this. could someone help here. (is it not wise to use structured streaming)

Re: Reg: Capacity consumption for fabric Notebook

Anonymous — Thu, 19 Dec 2024 02:27:50 GMT

Hi Mathan1787,

I'd like to suggest you check the official document about capacity throttling if helps for your scenario:

Understand your Fabric capacity throttling - Microsoft Fabric | Microsoft Learn

Regards,
Xiaoxin Sheng

Re: Reg: Capacity consumption for fabric Notebook

Anonymous — Wed, 25 Dec 2024 06:21:06 GMT

Hi Mathan1787 ,

Any update on this? Did the above suggestions help with your scenario? if that is the case, you can consider Kudo or Accept the helpful suggestions to help others who faced similar requirements.

Regards,

Xiaoxin Sheng

Re: Reg: Capacity consumption for fabric Notebook

Ray_Minds — Mon, 30 Dec 2024 10:06:05 GMT

Hi Mathan1787

Here you will do two things.

First approach:- If possible to you , you will the increase your capacity. F64 to another license.
https://learn.microsoft.com/en-us/fabric/enterprise/licenses
Second approach:- You must be check your all jobs scheduled timing. it should be different to each other. it will reduce the load. if you have issue with particular notebook. you need to do performance tuning for the particular notebook.

Please feel free to ping us if you need any help further.

If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.