Fabric Spark Run Series Analysis is a powerful observability feature designed to help you better understand, compare, and optimize your recurring Spark job executions.
Building on the momentum from preview and the announcement of the Autotune feature at the Fabric Conference, Run Series Analysis has expanded capabilities, improved accessibility, and enhanced engineering infrastructure to support enterprise-scale performance tuning.
What Is Spark Run Series Analysis?
Spark Run Series Analysis intelligently groups Spark application runs—originating from recurring pipeline activities, Notebook executions, Spark Job Definitions (SJDs), and Autotune-enabled runs—into automatically identified run series.
- Enabling users to:
- Detect outliers and anomalies across recurring runs.
- Understand changes in execution time and data inputs/outputs.
- Evaluate the impact of Autotune recommendations.
- Review detailed Spark SQL query configurations and performance breakdowns over time.
This feature provides a holistic view of Spark job behavior over time, helping users identify inefficiencies, troubleshoot regressions, and optimize execution performance.
Key Capabilities
Run Series Comparison - Compares the execution duration of a Spark run against historical runs within the same series. Drill into differences in input/output data to identify causes of performance variation.
Outlier Detection and Analysis - Automatically detect anomalous runs within a series and surface potential contributing factors such as resource constraints or configuration changes.
Detailed Run Instance View - Clicking into a single run instance reveals detailed time distribution metrics, providing insights into each phase of execution and surfacing opportunities for optimization. Configuration values used in the run—including those auto-tuned—are also displayed for reference.
Support for Running Applications - Run Series Analysis is now available even for Spark applications that are still in progress, offering earlier insights during runtime.
Access Points
You can access Spark Run Series Analysis from several entry points across the Fabric platform:
- Monitoring Hub’s Historical View.
- Recent Runs Panel in Notebooks or Spark Job Definitions.
- Spark Application Monitoring Detail Page.
Start Tuning Smarter
With the GA release of Spark Run Series Analysis, performance tuning in Microsoft Fabric becomes more proactive, data-driven, and insightful. Whether you're investigating anomalies, comparing runtime trends, or evaluating the impact of Autotune, Run Series Analysis equips you with the tools to drive efficiency at scale.
For more information, refer to the Monitor Apache Spark run series documentation.