topic Re: Fabric capacity issue - Python notebooks in Data Engineering

Fabric capacity issue - Python notebooks

fabric_1 — Fri, 18 Apr 2025 19:19:39 GMT

HI, We are using F8 capacity, we are have requirement of running 3 jobs once in every 5 mins, so the issue we are facing is capacity limit issue and we are unable to do anything once the limit is reached. The data volume is very low for all the three jobs so we are using python Notebooks instead of pyspark. Is increasing the capacity the only option we have?

We have a requirement to pull the data from 3 different APIs every 5 mins, currently we are using python script to pull the data and load it to lakehouse table. Each job is taking around a minute to complete. How do we make sure we don't reach the limits ? Is there any other alternate cost effective solution to achive this?

Re: Fabric capacity issue - Python notebooks

lbendlin — Sat, 19 Apr 2025 01:03:36 GMT

5 minutes is the sweet/sour spot. You are keeping your pools warm at all times and don't give your capacity an opportunity to cool down. Either stretch your interval (and accept startup delays) or go for a better capacity.

Re: Fabric capacity issue - Python notebooks

ObungiNiels — Sun, 20 Apr 2025 15:49:00 GMT

Hi @fabric_1 ,

I understand you have 3 separate jobs running every 5 mins, however I am wondering how these jobs are orchestrated. Do you run these from a pipeline or do you have 3 separate schedules on the notebooks themselves in place? Can you shed some light on that? 🙂

Kind regards,

Niels

Re: Fabric capacity issue - Python notebooks

Kar_c — Sun, 20 Apr 2025 18:57:48 GMT

hi @ObungiNiels, I have 3 separate pipelines running one after the other.

Re: Fabric capacity issue - Python notebooks

ObungiNiels — Tue, 06 May 2025 10:11:36 GMT

In this scenario, we can be sure that each notebook requires its own session for running. A python notebook runs on a 2 VCore, 16 GB RAM single node cluster. Having three of these running around the clock can consume a good chunk of your capacity.

I think it's worth it to reconsider the orchestration with the following things in mind:

- Running all three notebook from a single pipeline with High Concurrency for pipelines enabled in the Spark settings will cause all three notebook being run in the same session, limiting the amount of blocked VCores to what is consumed during a single session

- Alternatively, you can create a fourth ("control") notebook which leverages the runMultiple() function of notebookutils library to run all three notebooks within the same session, orchestrated from a notebook. The control notebook is then triggered from a data pipeline

Both of these option would result in your session being handed from one notebook to another rather than having all three notebook use their own. Implementing one of these options could help reduce the bound resources of your capacity.

Hope this helps! 🙂

Re: Fabric capacity issue - Python notebooks

Kar_c — Mon, 21 Apr 2025 11:52:13 GMT

Thanks @ObungiNiels. I will try these and update.

Re: Fabric capacity issue - Python notebooks

v-prasare — Thu, 24 Apr 2025 04:46:34 GMT

Hi @fabric_1,
Thanks for actively participating in MS Fabric community support

As we haven’t heard back from you, we wanted to kindly follow up to check if the solution provided for your issue worked? or let us know if you need any further assistance here?

Thanks,

Prashanth Are

MS Fabric community support

If this post helps, then please consider Accept it as the solution to help the other members find it more quickly and give Kudos if helped you resolve your query

Re: Fabric capacity issue - Python notebooks

v-prasare — Mon, 28 Apr 2025 09:38:13 GMT

Hi @fabric_1,
Thanks for actively participating in MS Fabric community support

As we haven’t heard back from you, we wanted to kindly follow up to check if the solution provided for your issue worked? or let us know if you need any further assistance here?

Thanks,

Prashanth Are

MS Fabric community support

If this post helps, then please consider Accept it as the solution to help the other members find it more quickly and give Kudos if helped you resolve your query

Re: Fabric capacity issue - Python notebooks

v-prasare — Mon, 05 May 2025 06:41:24 GMT

Hi @fabric_1,

As we haven’t heard back from you, we wanted to kindly follow up to check if the solution provided for your issue worked? or let us know if you need any further assistance here?

Thanks,

Prashanth Are

MS Fabric community support

If this post helps, then please consider Accept it as the solution to help the other members find it more quickly and give Kudos if helped you resolve your query