Gain Deeper Insights into Spark Jobs with JobInsight in Microsoft Fabric

[jejiang]

JobInsight is a powerful Java-based diagnostic library designed to help developers and data engineers analyze completed Spark applications in Microsoft Fabric. With JobInsight, you can programmatically access Spark execution metrics and logs—all from within a Fabric Notebook.

Whether you're investigating performance bottlenecks, debugging task execution, or conducting post-run analysis across jobs, stages, or executors, JobInsight surfaces valuable insights quickly and efficiently.

What is JobInsight?

JobInsight provides two core capabilities:

• Interactive Spark Job Analysis
  Offers structured APIs that return execution data—such as Spark queries, jobs, stages, tasks, and executors—as Spark Datasets for deep-dive analysis.
• Spark Event Log Access
  Allows users to copy Spark event logs to a OneLake or ADLS Gen2 directory for long-term storage or custom offline diagnostics.

With these tools, you can analyze, debug, and monitor Spark applications—all within your Fabric workspace.

Key Features and How to Use Them

Analyze Completed Spark Applications

You can extract critical execution data with just a few lines of code:

import com.microsoft.jobinsight.diagnostic.SparkDiagnostic

val jobInsight = SparkDiagnostic.analyze(
  workspaceId,
  artifactId,
  livyId,
  jobType,         // e.g., "sessions" or "batches"
  stateStorePath,  // Output path to store analysis results
  attemptId        // Optional; defaults to 1
)

val queries = jobInsight.queries
val jobs = jobInsight.jobs
val stages = jobInsight.stages
val tasks = jobInsight.tasks
val executors = jobInsight.executors

You can then apply standard Spark operations to explore trends, detect anomalies, and optimize performance.

Reuse Past Analyses

No need to rerun diagnostics on the same job—just reload previously saved results:

val jobInsight = SparkDiagnostic.loadJobInsight(stateStorePath)

val queries = jobInsight.queries
val jobs = jobInsight.jobs
// and so on...

This makes historical analysis and iterative debugging easy and efficient.

Save Metrics and Logs to a Lakehouse

You can persist analysis outputs into Lakehouse tables for reporting or integration:

val df = jobInsight.queries

df.write
  .format("delta")
  .mode("overwrite")
  .saveAsTable("sparkdiagnostic_lh.Queries")

Apply the same logic for other datasets such as jobs, stages, or executors.

Copy Event Logs to Lakehouse or ADLS Gen2

You can also copy event logs for deeper inspection or long-term retention:

import com.microsoft.jobinsight.diagnostic.LogUtils

val contentLength = LogUtils.copyEventLog(
  workspaceId,
  artifactId,
  livyId,
  jobType,
  targetDirectory,
  asyncMode = true, // Use async mode for best performance
  attemptId = 1
)

Example

val lakehouseBaseDir = "abfss://<workspace>@<onelake>/Files/eventlog/0513"
val jobType = "sessions"

copyEventLogs(
  workspaceId,
  artifactId,
  livyId,
  jobType,
  attemptId = 1,
  asyncMode = true,
  s"$lakehouseBaseDir/$jobType/async"
)

copyEventLogs(
  workspaceId,
  artifactId,
  livyId,
  jobType,
  attemptId = 1,
  asyncMode = false,
  s"$lakehouseBaseDir/$jobType/sync"
)

Use this method when you want to store raw logs outside the UI for further analysis.

Get Started

JobInsight makes Spark diagnostics in Microsoft Fabric easier, faster, and more powerful. By integrating with familiar Spark APIs and offering deep access to execution logs and metrics, it empowers you to:

• Visualize execution breakdowns.
• Monitor and tune resource usage.
• Identify and troubleshoot performance bottlenecks.
• Reuse and automate reproducible diagnostics.

For full documentation, check out: Job insight diagnostics library (Preview).