Overview

Every data engineer knows the drill. A Spark notebook fails at 3 AM. You open Fabric, navigate to the workspace, find the run, check the Spark Monitor, scan executor logs, cross-reference Spark Advisor — and end up flipping between tabs trying to piece it all together. There’s a simpler way with the Fabric Spark Operations Skill.

Meet the Fabric Spark Operations Skill

The Fabric Spark Operations Skill brings AI-assisted, read-only diagnostics for Spark workloads in Microsoft Fabric. Describe what you need in plain English and the skill resolves workspace and item IDs, queries Spark Monitoring APIs, and delivers a severity-ranked diagnostic report — without manual log parsing or portal navigation.

Example commands: Analyze Notebook runs

An example of the output follows:

Figure: Example diagnostic output from the Fabric Spark Operations Skill for notebook run.

Supported scenarios

Whether you’re digging into a single failure or trying to spot broader issues across Items or a workspace, Spark troubleshooting can take a lot of different forms. The following list covers some of the most common scenarios where the skill can help:

• Workspace health: Assess overall Spark activity, failure rates, and common issues across an entire workspace.
  Example prompt: “Review all Spark activity in workspace X”
• Failed notebook: Identify the root cause of a failed notebook run with severity-ranked findings.
  Example prompt: “Why did my notebook fail?”
• Pipeline failure: Trace failures through pipeline activities down to the underlying Spark session.
  Example prompt: “Diagnose pipeline run abc-123”
• Run history: Analyze patterns across past runs to detect recurring failures or regressions.
  Example prompt: “Review past runs of notebook X”
• Session triage: Check for stuck, zombie, or unhealthy Livy sessions consuming cluster capacity.
  Example prompt: “Check Livy session health”
• Performance: Surface skew, memory pressure, and compute sizing issues in completed runs.
  Example prompt: “Diagnose Spark performance for notebook X”
• Lakehouse jobs: Diagnose failures in lakehouse table maintenance and load operations.
  Example prompt: “Diagnose lakehouse table load failures”

How it works

Under the hood, the skill pulls data from Spark monitoring, logs, pipeline runs, and session activity to build a single report with specific fix recommendations and direct links back to the Fabric UI.

1. Tell it what to diagnose. Provide a notebook name, workspace name, or paste a URL. No GUIDs or portal navigation is required.
2. It finds the right run. The skill identifies the relevant run for you or asks a single clarifying question if needed.
3. Get the root cause. It goes beyond the stack trace to explain what went wrong and why, using the strongest available signal first.
4. Spot performance issues. Skew, memory pressure, and inefficiencies are detected and explained instead of left for you to hunt through stage metrics.
5. Check cluster sizing. The skill shows whether your cluster was under-resourced, over-utilized, or wasting capacity, with guidance on what to change.
6. Catch stuck sessions. It identifies sessions that are blocking capacity or hanging silently before they affect other workloads.
7. Review the results. Findings are ranked by severity with specific fix recommendations and a direct link to inspect the failed run in Fabric.

For deeper analysis, the skill can also copy event logs to OneLake and start a local Spark History Server container. This offers the full Spark UI with DAG visualization, SQL query plans, and task-level metrics, without needing portal access.

What makes this different

Answers, not raw logs. You get a plain-English explanation of what went wrong and what to fix, instead of a wall of stack traces to interpret yourself.

Works when data expires. Spark session data is retained for up to 30 days. If a run has aged out, the skill falls back automatically and still surfaces what it can, including the Failed Notebook snapshot and a direct link to inspect it in Fabric.

One view across pipelines and Spark. When a pipeline fails, you shouldn’t have to cross-reference three tools to find the cause. The skill traces the full chain from pipeline activity through the Spark session and tells you where things broke down.

Get started

The Fabric Spark Operations Skill is open-source and part of the skills-for-fabric toolkit on GitHub. To learn more about Fabric’s built-in Spark monitoring experiences, refer to the public Apache Spark monitoring overview, Spark monitoring APIs overview, and Spark monitoring best practice documentation.

To get started, install the skill and authenticate using az login. It works with GitHub Copilot, VS Code, Claude, and other AI coding assistants. See the README for full setup instructions.

Questions, issues, or feature requests? Open an issue on GitHub — contributions and feedback are welcome.