Forum Discussion

CReportify's avatar
CReportify
Frequent Visitor
1 year ago
Solved

Optimizing CU Usage in Microsoft Fabric

Hello everyone, I recently transitioned to using Microsoft Fabric products, specifically Notebooks (PySpark) and Lakehouse, running on F2 capacity. My current setup involves retrieving data from an ...
admin
capacities
  • nilendraFabric's avatar
    nilendraFabric
    1 year ago

    Hi CReportify 

     

    Yes, it is possible to reduce the driver cores when executing a notebook in Microsoft Fabric. You can configure the driver cores using the `%%configure` magic command at the beginning of your notebook. Here’s an example:

    %%configure -f
    {
    "driverMemory": "56g",
    "driverCores": 32,
    "executorMemory": "28g",
    "executorCores": 4
    }


    Microsoft Fabric does not currently offer a built-in way to manually trigger or schedule specific background tasks like metadata processing during off-peak hours. The platform automatically manages certain background processes, and there’s no direct control over when these tasks execute after capacity is resumed.

     

    Microsoft Fabric supports autoscaling for Spark pools. When you set up a Spark pool, you can enable autoscaling by configuring a range for the number of nodes. The system will automatically add or remove nodes based on workload demand.

    I have tried to answer all your queries here. 

    please accept this as solution if this resolved your query.

     

    thanks

     

     

v-sgandrathi's avatar
v-sgandrathi
Icon for Community Support rankCommunity Support
1 year ago

Hi CReportify,

Thankyou for connecting with Microsoft Community Forum.

 

Implement batch API calls, reduce driver cores, and optimize Spark processing to lower CU consumption. Utilize Fabric Pipelines for API-based ingestion to enhance efficiency, provide retry mechanisms, and improve concurrency control.

For frequent queries (e.g., Customer 2), consider transitioning to a Warehouse to boost performance. Continue using the Lakehouse for cost-effective bulk storage and processing.

To avoid CU spikes after capacity resumes, stagger workloads and monitor usage patterns for further adjustments.

These steps are expected to reduce costs while maintaining performance.

 

Please let us know if further assistance is required. Should this information prove useful, kindly mark it as a solution and consider giving us a Kudos.

 

Regards,
Sahasra.

  • v-sgandrathi's avatar
    v-sgandrathi
    Icon for Community Support rankCommunity Support
    1 year ago

    Hi CReportify,

     

    I wanted to check in on your situation regarding the issue. Have you resolved it? If you have, please consider marking the reply that helped you or sharing your solution. It would be greatly appreciated by others in the community who may have the same question.

     

    Thanks for using Microsoft Fabric Community Forum.