Hi @cgroleau , Thank you for reaching out to the Microsoft Community Forum.
If you're looking for a reliable, centralized way to collect and correlate logs from Spark notebooks in Microsoft Fabric, use the Fabric Apache Spark Diagnostic Emitter. This feature allows you to automatically send driver logs, executor logs, event logs and Spark metrics to destinations like Azure Log Analytics, Azure Event Hubs or Azure Storage. Once configured, it eliminates the need to manually review notebook snapshots by centralizing all logs in one place, enriched with metadata like ApplicationId, NotebookPath and TimeGenerated, so you can easily trace issues across notebook runs.
The best option for most teams is to send logs to Azure Log Analytics, where you can use Kusto Query Language (KQL) to search, analyze and correlate logs. To do this, create a Fabric Environment in your workspace settings. Then, add the necessary Spark properties to enable Log Analytics integration, pointing to your Azure Log Analytics workspace and securely referencing the workspace key through Azure Key Vault. After saving the environment, attach it to any notebook via the Environment dropdown in the notebook’s Home tab. From that point on, every Spark session will automatically emit logs to Log Analytics without additional code. Once the logs are flowing, you can query them from the Azure Portal using KQL. This gives you a real-time view of which notebooks are failing and how often, with the ability to drill into exact messages and timestamps.
If real-time monitoring is important to your workflow, you can configure the Diagnostic Emitter to send logs to Azure Event Hubs instead. These logs can then be streamed into a Fabric Eventstream, stored in a KQL Database (Eventhouse) and queried live with the same KQL language. This setup gives you immediate visibility into running jobs and failures, useful for building alerting pipelines or dashboards.
For teams that want to include custom application-level log messages like specific business rule checks or transformation steps, you can use Log4j directly inside your notebooks. These logs are automatically picked up by the Diagnostic Emitter, so they appear in Log Analytics or Eventstream alongside system logs.
If this helped solve the issue, please consider marking it “Accept as Solution” and giving a ‘Kudos’ so others with similar queries may find it more easily. If not, please share the details, always happy to help.
Thank you.
