Forum Discussion

CReportify's avatar
CReportify
Frequent Visitor
1 year ago
Solved

Optimizing CU Usage in Microsoft Fabric

Hello everyone, I recently transitioned to using Microsoft Fabric products, specifically Notebooks (PySpark) and Lakehouse, running on F2 capacity. My current setup involves retrieving data from an ...
admin
capacities
  • nilendraFabric's avatar
    nilendraFabric
    1 year ago

    Hi CReportify 

     

    Yes, it is possible to reduce the driver cores when executing a notebook in Microsoft Fabric. You can configure the driver cores using the `%%configure` magic command at the beginning of your notebook. Here’s an example:

    %%configure -f
    {
    "driverMemory": "56g",
    "driverCores": 32,
    "executorMemory": "28g",
    "executorCores": 4
    }


    Microsoft Fabric does not currently offer a built-in way to manually trigger or schedule specific background tasks like metadata processing during off-peak hours. The platform automatically manages certain background processes, and there’s no direct control over when these tasks execute after capacity is resumed.

     

    Microsoft Fabric supports autoscaling for Spark pools. When you set up a Spark pool, you can enable autoscaling by configuring a range for the number of nodes. The system will automatically add or remove nodes based on workload demand.

    I have tried to answer all your queries here. 

    please accept this as solution if this resolved your query.

     

    thanks

     

     

CReportify's avatar
CReportify
Frequent Visitor
1 year ago

Hi nilendraFabric,

 

Thanks for your response, its great!

 

Is there an option to reduce the driver cores when executing a notebook? Also, how to implement autoscalling for more dynamic load, do you have an example? Lastly, Is there a way to manually trigger or schedule specific background tasks (like metadata processing) to run during off-peak hours, so they don’t immediately execute after capacity is resumed?

 

Thanks in advance!

nilendraFabric's avatar
nilendraFabric
Icon for Super User rankSuper User
1 year ago

Hi CReportify 

 

Yes, it is possible to reduce the driver cores when executing a notebook in Microsoft Fabric. You can configure the driver cores using the `%%configure` magic command at the beginning of your notebook. Here’s an example:

%%configure -f
{
"driverMemory": "56g",
"driverCores": 32,
"executorMemory": "28g",
"executorCores": 4
}


Microsoft Fabric does not currently offer a built-in way to manually trigger or schedule specific background tasks like metadata processing during off-peak hours. The platform automatically manages certain background processes, and there’s no direct control over when these tasks execute after capacity is resumed.

 

Microsoft Fabric supports autoscaling for Spark pools. When you set up a Spark pool, you can enable autoscaling by configuring a range for the number of nodes. The system will automatically add or remove nodes based on workload demand.

I have tried to answer all your queries here. 

please accept this as solution if this resolved your query.

 

thanks