Custom Spark Environment Missing Startup Logs in Microsoft Fabric Notebooks

Hi HoneHealthMB ,
Thanks for reaching out to the Microsoft fabric community forum

Thanks for the detailed explanation — this is a great question and you've already done a solid job narrowing down the issue. Based on your description, it sounds like your custom Spark environment in Microsoft Fabric is not emitting early lifecycle logs because it bypasses some of the Fabric-specific bootstrap mechanisms.

Let me walk through what’s likely happening and how to resolve it.

1. What’s Responsible for Emitting Startup Logs?

In Microsoft Fabric, the early logs (like SAS token retrieval, Spark context injection, ZooKeeper init, etc.) are not emitted by Spark directly. Instead, they come from Fabric’s internal runtime orchestration, which includes:

• Entrypoint scripts in the default Spark Docker image
• Microsoft-injected Spark plugins
• Livy session warm-up logic
• Diagnostic emitters and redirect plugins

When using a custom environment, especially if it uses a different base image or overrides startup behavior, these components may not get initialized — resulting in missing logs in prelaunch.out, stderr-active, and stdout-active.

2. Why Are the Logs Missing in Your Custom Environment?

You've already identified a few causes:

• spark.livy.synapse.session-warmup.enabled is set to false
• Missing spark.plugins = org.apache.spark.microsoft.tools.api.plugin.MSToolsRedirectExecutorConsoleOutputPlugin
• Likely missing Fabric’s bootstrap scripts from the Docker image

All of these play a role in suppressing the log output normally visible in the default environment.

3.How to Fix This

Here’s how to bring back the diagnostics and logs in your custom Spark environment:

4. Enable Livy Session Warm-Up

You’ve noticed this is disabled:

spark.livy.synapse.session-warmup.enabled=false

This warm-up process initializes components that emit early diagnostics and environmental metadata. You should enable it by setting:

spark.livy.synapse.session-warmup.enabled=true

5. Add the Missing Spark Plugin

This plugin redirects executor and Spark context output to the notebook UI:

spark.plugins=org.apache.spark.microsoft.tools.api.plugin.MSToolsRedirectExecutorConsoleOutputPlugin

If not set, logs won’t be redirected properly, especially during executor startup.

6. Use the Recommended Microsoft Base Image (If Possible)

Microsoft’s default Spark environment in Fabric uses a custom base image that includes:

• Preconfigured diagnostic emitters
• Bootstrap scripts for token handling, ZooKeeper, etc.
• The proper entrypoint to emit logs

If you’re starting from a generic Spark base image (e.g., bitnami/spark or spark:latest), these components are missing.

7.Recommended action: Contact Microsoft support and request access or documentation for the appropriate Fabric-compatible Spark base image. They may provide one like:

bash

mcr.microsoft.com/synapse/spark

Or internal documentation on expected image structure.

8. Verify Entrypoint or Init Scripts

Ensure your Docker image preserves or replicates Microsoft’s entrypoint scripts, which typically emit:

• Environment variable metadata
• Resource setup logs
• Token and config resolution steps

If your custom image overrides the entrypoint with a simplified one, all of this early logging gets skipped.

9. Official Microsoft Docs for Reference:

Monitor Apache Spark applications with Azure Log Analytics - Microsoft Fabric | Microsoft Learn
Collect your Apache Spark applications logs and metrics using Azure Event Hubs - Microsoft Fabric | Microsoft Learn
Create, Configure, and Use an Environment in Fabric - Microsoft Fabric | Microsoft Learn

If you found this post helpful, please 'consider giving it Kudos' and marking it as the 'accepted solution' to assist other members in finding it more easily.

Thank you.