Forum Discussion

remconicolai-'s avatar
remconicolai-
Frequent Visitor
2 years ago
Solved

Increasing CU capacity usage running dataflows

I've scheduled just a few dataflows in Fabric to transfer data from parquet files to a Fabric lakehouse. While running this schedule every hour, the CU background usage exceeds and finaly reaches the 100% and the service crashes. The dataflows are not very complicated so I don't no why this happens and how to prevent it?

 

Example

Dataflow
  • Anonymous's avatar
    Anonymous
    2 years ago

    Hi, SaiTejaTalasila 

    thanks for your concern about this issue. 

    Your method is also a solution. 

    And I would like to share some additional solutions below.

     

    Hi,remconicolai- 

    I am glad to help you.  

    According to your description, you want to know what causes this exceeding CU growth for just a few simple dataflows? 

     

    If I understand you correctly, then you can refer to my solution. 

     

    Check the complexity of the data flow: Even if the data flow seems simple, make sure there is no hidden complexity or inefficiency. Some transformation operations may be more resource-intensive than expected, so you can check for transformations and optimize some of them appropriately. 

      

    Throttling Policies: Understand and configure throttling policies to manage and balance performance and reliability. 

      

    Monitor CU Usage: Use the  Microsoft Fabric Capacity Metrics app to monitor CU usage. This helps determine if there are specific times or operations that cause spikes. 

      

    Adjust Scheduling: Instead of running streams every hour, try staggering times or running streams during off-peak hours to distribute the load more evenly. 

      

    Increase Capacity: If your current capacity is consistently reaching its limits, consider increasing Fabric capacity to better handle the load. 

     

    Here's the official documentation, hope it helps: 

    Understand your Fabric capacity throttling - Microsoft Fabric | Microsoft Learn 

     

    I hope my suggestions give you good ideas, if you have any more questions, please clarify in a follow-up reply.


    Best Regards,
    Fen Ling,


    If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.

8 Replies

  • SaiTejaTalasila's avatar
    SaiTejaTalasila
    Super User
    2 years ago

    Hi remconicolai- ,

     

    Instead of dataflow gen2 you can try to use notebooks spark for the transformations/transfer.Then you will be able to limit the number of executes on workspace level.So,it will use limited resources for the task and avoid the CU issue.

     

    What is the source of your parquet files?

     

     

    Thanks,

    Sai Teja 

  • Anonymous's avatar
    Anonymous
    Not applicable
    2 years ago

    Hi, SaiTejaTalasila 

    thanks for your concern about this issue. 

    Your method is also a solution. 

    And I would like to share some additional solutions below.

     

    Hi,remconicolai- 

    I am glad to help you.  

    According to your description, you want to know what causes this exceeding CU growth for just a few simple dataflows? 

     

    If I understand you correctly, then you can refer to my solution. 

     

    Check the complexity of the data flow: Even if the data flow seems simple, make sure there is no hidden complexity or inefficiency. Some transformation operations may be more resource-intensive than expected, so you can check for transformations and optimize some of them appropriately. 

      

    Throttling Policies: Understand and configure throttling policies to manage and balance performance and reliability. 

      

    Monitor CU Usage: Use the  Microsoft Fabric Capacity Metrics app to monitor CU usage. This helps determine if there are specific times or operations that cause spikes. 

      

    Adjust Scheduling: Instead of running streams every hour, try staggering times or running streams during off-peak hours to distribute the load more evenly. 

      

    Increase Capacity: If your current capacity is consistently reaching its limits, consider increasing Fabric capacity to better handle the load. 

     

    Here's the official documentation, hope it helps: 

    Understand your Fabric capacity throttling - Microsoft Fabric | Microsoft Learn 

     

    I hope my suggestions give you good ideas, if you have any more questions, please clarify in a follow-up reply.


    Best Regards,
    Fen Ling,


    If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.