As Spark workloads scale in size and complexity, fast access to Spark application execution metrics and logs is essential for debugging, performance analysis, and operational confidence. The Spark History Server is a key part of that workflow — but for large workloads, loading times have been a significant pain point.
With this release, we’ve introduced a new snapshot-based approach that significantly improves Spark History Server loading performance in Microsoft Fabric.
The challenge
Previously, the Spark History Server had to fully parse and render the entire application event log before the UI could load. For large-scale enterprise workloads — applications with hundreds of thousands of tasks, event logs exceeding 10 GB, or long-running Spark Streaming jobs — this could take up to an hour, or in some cases, fail to load entirely.
What’s new
The Spark History Server now uses a snapshot-based approach to load execution metrics and logs incrementally from the backend, rather than processing the full event log upfront. Users can see the first batch of data quickly, then load additional details on demand — resulting in a much-improved experience.
This means applications that previously took up to an hour to load now render in seconds or minutes, unblocking enterprise customers running complex batch jobs and streaming pipelines at scale.
Figure: Performance comparison — Improved Spark History Server (left) vs. original (right).
Expanded Spark Streaming support
We’ve also removed the restriction that prevented users from enabling the spark.eventLog.compress and spark.eventLog.rolling.enabled properties. Previously, enabling these configurations would cause the Spark History Server to fail to render. Users can now enable event log compression and rolling for Spark Streaming and other long-running applications, and the History Server will continue to render properly.
Learn more
These improvements make the Spark History Server faster, more scalable, and more reliable for workloads of any size. To learn more about Spark History Server in Microsoft Fabric, refer to the Fabric Spark documentation.
These improvements make the Spark History Server faster, more scalable, and more reliable. To learn more about Spark History Server in Microsoft Fabric, refer to the Spark History Server documentation for Microsoft Fabric.
Microsoft Employee