The Spark Applications Comparison feature is now in preview in Microsoft Fabric. This new capability empowers developers and data engineers to analyze, debug, and optimize Spark performance across multiple application runs—whether you're tracking changes from code updates or data variations to improve performance.
What is Spark Applications Comparison?
The Spark Applications Comparison feature allows users to select and compare up to four Spark application run side by side. By visualizing and contrasting key execution metrics, users can more easily identify performance regressions, improvements, and anomalies.
Key Use Cases
- Compare runs from the same artifact (e.g., Notebook or Spark Job Definition).
- Identify regressions or improvements by viewing metric deltas from a base run.
- Debug performance issues using execution time, input/output data trends, and resource utilization metrics.
Explore the Compare Panel
The Compare Panel provides a side-by-side breakdown of key performance and resource metrics. Each Spark application run can be easily compared to the base run, with clear indicators of change and percentage variation
Metrics include
- Start Time
- Running Duration
- Queued Duration
- Total Duration
- Executor CPU Time & Run Time
- Input / Output Size and Record Counts
- Disk & Memory Spill
- Shuffle Write Volume and Record Count
For anomalous runs, the panel also surfaces potential root causes and highlights time-intensive activities to help focus your debugging efforts.
Deep Dive: Spark Monitoring L2 Integration
For any run in the comparison view, you can navigate to the detailed Spark L2 Monitoring page to explore jobs, stages, logs, and configuration history in greater depth—enabling root cause analysis and fine-tuned performance optimization.
How It Works
Access the Compare Feature - You can launch the comparison view directly from the Monitor Run Series page:
- Navigate to the Run Series page.
- Switch to the Compare Runs tab.
- Select up to four completed runs to compare. One run will serve as the base run for reference, and others will be compared against it.
You can reassign the base run at any time by clicking the flag icon.
Get Started
To try out the Spark Applications Comparison feature, head over to the Monitor Run Series page in Microsoft Fabric. Select a few completed runs, open the Compare Panel, and start exploring how your Spark applications evolve over time.
For more information, refer to the documentation for Apache Spark Applications Comparison.