Forum Discussion
Increasing CU capacity usage running dataflows
- Anonymous2 years ago
Hi, SaiTejaTalasila
thanks for your concern about this issue.
Your method is also a solution.
And I would like to share some additional solutions below.
I am glad to help you.
According to your description, you want to know what causes this exceeding CU growth for just a few simple dataflows?
If I understand you correctly, then you can refer to my solution.
Check the complexity of the data flow: Even if the data flow seems simple, make sure there is no hidden complexity or inefficiency. Some transformation operations may be more resource-intensive than expected, so you can check for transformations and optimize some of them appropriately.
Throttling Policies: Understand and configure throttling policies to manage and balance performance and reliability.
Monitor CU Usage: Use the Microsoft Fabric Capacity Metrics app to monitor CU usage. This helps determine if there are specific times or operations that cause spikes.
Adjust Scheduling: Instead of running streams every hour, try staggering times or running streams during off-peak hours to distribute the load more evenly.
Increase Capacity: If your current capacity is consistently reaching its limits, consider increasing Fabric capacity to better handle the load.
Here's the official documentation, hope it helps:
Understand your Fabric capacity throttling - Microsoft Fabric | Microsoft Learn
I hope my suggestions give you good ideas, if you have any more questions, please clarify in a follow-up reply.
Best Regards,
Fen Ling,
If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.
Hi remconicolai- ,
Instead of dataflow gen2 you can try to use notebooks spark for the transformations/transfer.Then you will be able to limit the number of executes on workspace level.So,it will use limited resources for the task and avoid the CU issue.
What is the source of your parquet files?
Thanks,
Sai Teja
The source is Azure storage, blob. I'd like a solution for the dataflows and not to notebooks instead.
- SaiTejaTalasila2 years agoSuper User
Hi remconicolai- ,
As per my knowledge on notebooks it gives more flexibility on resource utilisation but on power query as per my understanding it's not possible to limit the resources.On tenant level you can set query limitations, but it applies for entire tenant.
https://microsoftlearning.github.io/mslearn-fabric/Instructions/Labs/10-ingest-notebooks.html
Thanks,
Sai Teja
- remconicolai-2 years agoFrequent Visitor
Still my question is, what causes this exceeding CU growth for just a few simple dataflows?
- lognvik1 year agoFrequent Visitor
We're experiensing the same insane usage for basically nothing. Messy notebooks are not a viable option for now. Any progress on this on your side?