Skip to main content
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 

Data Days is here! Join us now for 60+ days of learning, challenges, and connection. Learn more

varunjain
‎11-20-2024 09:30 AM

Announcing preview of Workspace Monitoring

Overview

Workspace monitoring allows Fabric developers and admins to access detailed logs and performance metrics for their workspaces. This helps troubleshoot performance issues, investigate errors, optimize queries, and minimize data downtime. Workspace monitoring provides visibility into operations crucial for observing system health, resource optimization, and operational planning. Insights derived from this data can improve performance tuning, capacity planning, resource allocation, and decision-making, while also aiding compliance and auditing by maintaining detailed logs.

Why Workspace Monitoring?

In today’s data-driven world, real-time insights and diagnostics are essential to ensure the smooth operation of your applications and data services. With workspace monitoring, we’re putting the power in your hands.

Now, you can access a monitoring experience that allows you to:

  • Access critical data on demand: Query diagnostic logs and performance metrics directly within your Fabric environment, giving you instant access to the information you need, when you need it.
  • Troubleshoot with confidence: Identify and resolve issues quickly by accessing granular logs and performance metrics, right from your own monitoring solution.
  • Build customized monitoring dashboards: With the monitoring database, you can integrate data into your preferred tools like Power BI and Real-Time Dashboards to create dashboards tailored to your needs.
  • Set up alerting: Set up alerts based on the logs and metrics you’re tracking.

Workspace monitoring allows you to collect and analyze logs for a variety of use cases. For example, the Semantic Model logs offer deep insights into your semantic model's performance and usage, allowing you to optimize operations at every level. With this information, you can track engine activity by capacity, workspace, and hour, observe daily or hourly engine loads, and identify operations that consume the most CPU time. User-generated load and query analysis are accessible, helping you see which DAX queries are issued, their execution costs, and performance details like time spent in the Storage Engine versus the Formula Engine. For refresh operations, you can identify costly or overlapping refreshes, examine parallel versus sequential tasks, and review each operation's duration and sub-steps, giving you a clear view of the resource impact across your model's lifecycle.

  • As a BI developer authoring semantic models and publishing to the service, you can dive into the Semantic Model logs to understand query performance, look into refresh history, understand DAX query patterns, etc.  
  • As an Eventhouse developer building real-time intelligence solutions, you can access ingestion, queries, performance metrics, etc. for understanding and optimizing Eventhouse performance.
  • As a Fabric capacity administrator assessing capacity health and availability, you can query logs to evaluate individual query durations and identify optimization scenarios, including better capacity management opportunities like scheduling refreshes, etc.

Key Features of Workspace Monitoring

  • Monitoring Database: Read-only monitoring Eventhouse to surface logs and metrics from Fabric workloads.
  • End-to-end visibility: Consistent, standardized troubleshooting experience across supported Fabric items.
  • Log analysis: Analyze historical logs and detect anomalies with KQL queries and time-series analysis.
  • Detailed diagnostic logs: Access granular logs and metrics including query text, operation details, CPU consumption, etc.

Supported Fabric Workloads

This feature currently supports the listed workloads and tables, more workloads will be added incrementally. See Workspace Minitoring overview more details on the monitoring scenarios, log tables and schema.

  • Power BI - Analysis Services Engine
    • SemanticModelLogs - logs Analysis Services trace events like Command, DirectQuery, Error, ExecutionMetrics, etc.
  • Real-time Intelligence - Eventhouse
    • EventhouseQueryLogs – logs all Eventhouse KQL queries.  
    • EventhouseCommandLogs – logs all Eventhouse commands. 
    • EventhouseDataOperations – logs all successful data operations including Batch ingestions, Streaming seal operations (operations that store streaming data to database extents), Materialized views updates, and Update policy table updates. 
    • EventhouseIngestionResultLogs – logs all successful and failed ingestions, including the file path of the ingested data.
    • EventhouseMetrics- set of metrics that provide in-depth monitoring of ingestions, materialized views, and continuous exports.
  • Data Engineering - API for GraphQL
    • GraphQLLog: query logs from queries run by the Fabric API for GraphQL on its connected data sources.
    • GraphQLMetrics: metrics for understanding query performance and trends.

How Workspace Monitoring Works

When workspace admins enable monitoring for their workspaces, the system creates a Monitoring Eventhouse and Monitoring Event stream within the workspace. These Fabric-native items are read-only items that are used to surface logs and metrics from the workspace into a dedicated Kusto database within the Eventhouse.

The Monitoring Eventstream is read-only and cannot be edited to add destination or change parameters. Similarly, the Monitoring Eventhouse is a read-only Fabric item and cannot be re-shared directly. You can always share this Eventhouse with additional users by adding them to the workspace roles. The logs are sent continuously and typically arrive within 10-15 minutes of the activity being generated in the item.

Important Note: Workspace monitoring will be available at no additional cost during preview in the near term, but billing and consumption usage will be announced soon. Stay tuned to the Fabric blog channel for updates

Workspace Monitoring Demo

Get Started Today!

To get started today, we strongly encourage Fabric administrators to configure monitoring tailored to their specific needs. The tenant setting is turned on by default and is automatically delegated to capacity admins. Note, capacity admins can over-ride tenant admin configuration given this setting is auto delegated. However, workspace admins must still enable it within their individual workspaces to access this capability.

Admin Controls for Workspace Monitoring

Take advantage of this feature by:

  • Trying it out in your tenant and building custom queries or utilizing sample queries provided in the documentation.
  • Enabling this feature within your workspace and collaborating with developers to proactively troubleshoot Fabric operations.
  • Querying these logs to better understand performance and configurations and potentially optimizing resource usage and performance for end users.

We encourage you to check out the links below to learn more about this new monitoring capability and try it out for yourself.

Resources

Workspace-Monitoring-Demo.mp4
Enabling-Workspace-Monitoring_TenantSettings.mp4