Forum Discussion
onelake other operations via redirect?
Hi
When i first set up Fabric dataflows which call rest API's and save incremental updates to azure sql db, the onelake cu was minimal. dataflow staging is turned off and we dont use the lakehouse at all for anything.
Fabric is constantly being throttled as the Lakehouse - DataflowsStagingLakehouse is consuming more CU than all the other activities in Fabric. Focusing on the lakehouse item, i see "onelake other operations via redirect" accounting for most of the CU usage, with other proxy read/writes accounting for the balance of the lakehouse CU.
I constantly see lakehouse CU usage whilst there are not dataflows running and randomly spiking 1000% and then going back to the average - when nothing else is running.
Are these lakehouse charges for running the dataflows? If so - then what are the dataflow CU charges for as "staging" is turned off for all dataflows and how are these 1000% lakehouse cu usages occuring when no dataflows are even running?
I have asked Microsoft and spoken to maybe 20 different agents - no one can explain the lakehouse cu usage and why the lakehouse is consuming cu when no dataflows are running (no its not smoothing or bursting)
Any direction here would be appreciated.
Hi shaunmw,
Thank you for reaching out to the microsoft Forum Community.
Could you please confirm if your query has been resolved? If so, kindly mark the helpful response as a solution. If you have any further questions, feel free to reach out to us.
If this helps then please Accept it as a solution and dropping a "Kudos" so other members can find it more easily.
Hope this works for you!
Thanks.
10 Replies
I have figured out what the issue is
I deleted all the tables in the lakehouse created by fabric and the excessive CU consumption stopped immediately.
So excessive lakehouse CU occurs when dataflows run and fabric created tables exist in the lakehouse - these tables are then refreshed aswell which consumes the excessive CU.
I just ran a python notebook to delete the thousands of tables which fabric created in the lakehouse - it did take several hourds to run and consumed 300,000 CU(s) in the process.
v-saisrao-msftCommunity Support
Hi shaunmw,
Thank you for the detailed explanation of your understanding. I am glad to hear that you seem to have figured out your issue. I would appreciate it if you could confirm whether the problem mentioned in the original query still persists or if it has been completely resolved.
Thank you.
- v-saisrao-msft
Hi shaunmw,
Thanks for sharing your findings! Glad to hear you’ve identified the cause. Could you confirm if the issue is fully resolved or if you're still seeing unexpected CU usage?
If the issue is resolved, kindly consider accepting your response as the solution to assist other Community Members.
Thank you.
- nilendraFabric
Super User
Hello shaunmw
These points might help you :
Operations labeled as “via Redirect” generally indicate optimized transactions where requests are routed efficiently to reduce compute consumption. However, these still consume CUs for tasks like metadata updates or system-triggered activities
https://github.com/MicrosoftDocs/fabric-docs/blob/main/docs/enterprise/fabric-operations.md
Even with staging turned off for Dataflows, the Lakehouse (e.g., DataflowsStagingLakehouse) might still be utilized for temporary storage or system operations. This could explain why Lakehouse CU usage persists or spikes despite no active dataflows running
The spikes could also result from system-triggered refreshes, indexing, or mirroring activities that Fabric performs autonomously
At no point of time I am suggesting that 1000% hike is justifiable by these points.
Are you pausing the capacity?
if yes schedule capacity pauses during low-usage periods to minimize unnecessary background activities. Be aware that resuming capacity can trigger spikes due to smoothing mechanisms
hope this is helpful
Hi Nilendra
if the lakehouse is used for temp storage or system operations - what is the staging data warehouse then used for since there is also CU usage for the staging datawarehouse aswell?
i just find it hard to believe that an activity that is not under a user control, we have no insight to what it is or how to manage it or what process is driving the usage, or how to optimize it, is the single largest CU consumer on the platform by about 60%.
capacity is not turned off at any stage.
these lakehouse high cu usages are pushing fabric into throttling on a regular basis.
What is strange is the lakehouse activity can occur when there is no dataflow activity, increase by 600% in 1 day or 1000% from 1 hour to the next with all the same dataflows and data sources in place.
- nilendraFabric
Hello shaunmw
This seems to be a bug and case has been created with MS.
please read this discussion
Quoting some part of it
"
I sent a example:
I have 0.625CU consumed for 24h, related to a pipeline which consumed 1800CU.
so this background activity will cost me 54 000CU per 24h for something which cost only 1800CU during its execution and was burndown in seconds.
0.625 * 60 * 60 * 24 = 54 000 CU in total
if the impact on my capacity is 0.625, this activity should consumed my capacity for 48 minutes
0.625 * 60 * 48 = 1800
else I expect, 1800 / (24 * 60 * 60) = 0.0208 CU consumed for 24h
- "
please accept this solution if it helps.
- v-saisrao-msft
Hi shaunmw,
Thank you for reaching out to the microsoft Forum Community.
Could you please confirm if your query has been resolved? If so, kindly mark the helpful response as a solution. If you have any further questions, feel free to reach out to us.
If this helps then please Accept it as a solution and dropping a "Kudos" so other members can find it more easily.
Hope this works for you!
Thanks.
- v-saisrao-msft
Hi shaunmw,
We haven’t heard back from you regarding your issue. If it has been resolved, please mark the helpful response as the solution and give a ‘Kudos’ to assist others. If you still need support, let us know.
Thank you.
Has there been any resolution? As far as I can tell, we are seeing a similar pattern. None of the replies above seem to address the actual issue.
Hi Jon
The problem was a fabric bug which created thousands of unrequired tables and queries in onelake dataflows staging for the dataflows, even though the 'staging' was unticked for the dataflow.
whenever the dataflows refreshed, these thousands of tables and queries in onelake then refreshed aswell which absolutely smashed our tenant CU.
After 20 calls with microsoft technical support, they were unable to provide any direction on which tables in onelake dataflows staging were required by fabric and which were not. I eventually gave up on getting an answer from microsoft and just wrote a python notebook to delete all tables in the dataflows staging. the notebook probably took 5 -6 hours to clear all the tables and after that the CU usage dropped off sharply and no more issues.
Just google / gpt how to do the python notebook if thats the same issue you have.
