Forum Discussion

Mathan1787's avatar
Mathan1787
Helper I
1 year ago

Reg: Capacity consumption for fabric Notebook

Hi,

 

I am recently trying out spark structured streaming in fabric notebook and try to check if its processing the files in real time. My notebook is running for past 2 days with very minimum vcore and Memory for both driver(1) and executor (1) via environment.   In my previous try, i didn't have environment and job ran for 3 days with f64 capacity and at one point capacity consumation is 100%. Capacity consumation is keep increasing for each day irrespective of new environment. if the job runs for a long time does it cost more CU. I dont really understand this. could someone help here. (is it not wise to use structured streaming)

 

 

 

data engineering
Environment
Spark Job Definition

3 Replies

  • Anonymous's avatar
    Anonymous
    Not applicable
    1 year ago

    Hi Mathan1787 ,

    Any update on this? Did the above suggestions help with your scenario? if that is the case, you can consider Kudo or Accept the helpful suggestions to help others who faced similar requirements.

    Regards,

    Xiaoxin Sheng

  • Ray_Minds's avatar
    Ray_Minds
    Solution Supplier
    1 year ago

    Hi Mathan1787 

    Here you will do two things.

    1. First approach:- If possible to you , you will the increase your capacity. F64 to another license.
      https://learn.microsoft.com/en-us/fabric/enterprise/licenses
    2. Second approach:- You must be check your all jobs scheduled timing. it should be different to each other. it will reduce the load. if you have issue with particular notebook. you need to do performance tuning for the particular notebook.

      Please feel free to ping us if you need any help further.


    If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.