Unexpected Error in Fabric Notebook When Displaying Lakehouse Table – Works After Restarting Session

Issue:
While running notebook code in Microsoft Fabric to display a Lakehouse table, an “Unexpected Error” appears. After stopping the current session and starting a new session, the same code runs successfully.

Root Cause:

Based on analysis, the issue is most likely related to Spark session instability or state corruption, not the code itself.

Possible technical reasons:

1. Spark session state corruption – The active Spark session may enter an inconsistent state.

2. Driver memory/resource exhaustion – Long-running sessions may consume memory or executor resources.

3. Idle timeout or session expiration – Session partially expires but UI still shows it as active.

4. Metadata/cache inconsistency – Cached table metadata may become stale.

5. Background cluster restart – Spark backend may restart but notebook session doesn’t fully reconnect.

Since restarting the session fixes the issue without code changes, it confirms the problem is session-level, not logic-level.

If you face this issue:

1. Stop the current notebook session.

2. Start a new Spark session.

3. Re-run the notebook.

Additional preventive checks:

• Avoid keeping sessions idle for long periods.

• Clear cache before re-running:  Code: spark.catalog.clearCache()

• Restart session after heavy transformations.

• Monitor Spark pool memory usage.

If the issue happens frequently:

• Check Spark pool auto-scale settings.

• Review runtime version compatibility.

• Open a Microsoft support ticket with session logs.

Thanks,
Hitakshi Chauhan

Fabric MSFabric community fabriccommunity

Great HitakshiChauhan  👍

A few additional tips for anyone facing this "Unexpected Error" in Fabric Notebooks:

1. Check your Spark session state proactively

# Quick health check before heavy operations
print(f"Spark version: {spark.version}")
print(f"Active session: {spark.sparkContext.applicationId}")

2. Use spark.catalog.refreshTable() before display
If you suspect metadata cache issues on a specific table:

spark.catalog.refreshTable("lakehouse_name.table_name")
display(df)

3. Avoid long idle sessions
Fabric Spark sessions can become unstable after extended idle periods. If you step away for 30+ min, it's often faster to restart the session than to debug random errors.

4. Monitor with Spark UI
Check the Spark UI (Monitoring Hub, Spark Application) for memory pressure or failed stages before restarting blindly.

Thanks for sharing this, it's a common pain point that many Fabric users encounter!