Workspace Monitoring - compute costs ?

Hi,

This is a general response to all above.

First, thanks all who responded. We were already aware of the various options mentioned but good to get further input from the community.

So thanks to…

deborshi_nag, strzala   for mentioning FUAM – we are considering it for other use cases, but need to confirm whether it also ingests job schedule runs for this particular use case.

AsgerLB  for providing personal observations of workspace monitoring consumption. And also for mentioning Azure Log analytics – the benefit of it not consuming capacity compute, of course means additional costs, so we were still looking at how to estimate that, then determine whether we can get that additional budget.
 

Regarding our own observations…

Firstly, on the Kratos BI video.

 The “red flag” in the video is the large “hump” in CU consumption due to Eventhouse/Eventstream visible in the ribbon graph. However, one annoying aspect of the ribbon graph is that it does not indicate what fraction of total available capacity is being shown. This means that while it is good for visualizing the relative contributions of the various workload types, it does nothing as to whether these are a significant % of capacity.

This means the large hump shown in the video may not really be a red flag unless it is a significant %. The real “red flag” in the video is his mention of receiving capacity alerts, as that (should) indicate significant levels of consumption.

Based on other details visible we reached the following conclusions:

• The capacity was an F64 (based on CU utilization and daily total)
• The capacity was being barely used (based on CU utilization on the right)
• The WM consumption in the ribbon graph represented barely 0.1% of capacity
• The reason it was such a significant proportion of the total usage in the ribbon graph was down to:
• There being very little usage in the first place
• Possibly also due to semantic model events likely being the majority of events, and for small / fast semantic model refreshes that could form a much larger “relative” monitoring overhead

Secondly, some items that might be useful for others based on our own investigations.

WM covers 5 areas:

• Fabric ( Jobs)
• Data engineering (GraphQL)
• Real time (Eventhouses – multiple logs)
• Mirrored databases
• Power BI (semantic models)

Our tests were on some medium sized production workspaces that would only be producing Jobs and Semantic Model events. 
So please be aware observations will not apply to workspaces with significant footprints in the other areas, and in particular we would expect workspaces with event houses to have significantly higher WM compute costs.

This was on an F64 capacity.

We observed:

• Jobs events were on the order of 100 per day.
• Semantic model events 50,000 per day.
• Event house uptime was ~2.5 hours per day.
• WM consumption per workspace was  ~0.6% of total capacity.
• We will likely use 1% as our “conservative” estimate.

There were also several things we noted about the WM feature. 

While the general interest of anyone enabling WM will be observability, realistically almost everyone will be interested in creating alerts.

• Because the event stream is read only you cannot use it to create alerts.
• You can create alerts (activators) from KQL queries in the event house, but:
• The activator for this is polled  and you have to set a polling frequency.
• Documentation notes that if you set this frequency to 5 minutes or less then the eventhouse will have 100% uptime.

The other area of interest would be longer term retention for historical analysis, trending of operational data.
Likely people would want to be moving it into a lakehouse or similar.

• Because the event house is read only you cannot enable one lake availability.
• Even if you could this would have the same retention as the event house.
• This means you will have to “roll your own” solution for long term retention, but lakehouse availability may have been useful as starting point for ingestion.

Also some observations on other options for job events.
If we exclude Azure Log analytics, there are 3 ways to get job event data :
(well actually 4 but we won’t talk about the 4th 😉 )

• Job Schedule REST API
• Fabric Job Events, (event source)
• Workspace Monitoring

Worth noting:

• Only the Job REST API includes failure reason (i.e. error message) in addition to the job status.
• Fabric Job Events and WM do not so if you wanted alerts to include this you would have to do some custom work.
• WM is the only one that get events at workspace scope.
• REST API and Job events are at item scope so if you want to get events for the entire workspace you need to do extra work.
• It is a bit annoying that there is no readily available source for jobs data at workspace scope, especially when the monitoring hub has already collected this data.
• unlike jobs there are no convenient alternative sources/APIS for the data in most other WM logs e.g . Semantic Models
  • So if you are interested in those then WM may be the only choice.
    
    

Hope this is of use to some people.

Cheers