Unexpected Error in Fabric Notebook When Displaying Lakehouse Table – Works After Restarting Session

[HitakshiChauhan]

Hi Community,

I am facing an issue while running code in a notebook in Microsoft Fabric to display a table from the Lakehouse.

Sometimes when I execute the code (for example, to display the table using Spark), I get an “Unexpected Error”. However, when I stop the current running session and start a new session, the same code works correctly without any changes.

I would like to understand the root cause of this behavior:

• Is this related to Spark session state?

• Could it be a cluster resource or timeout issue?

• Is there any caching or session conflict happening in the background?

Has anyone else experienced a similar issue? Any guidance on how to permanently resolve this instead of restarting the session each time would be helpful.

Thank you in advance for your support.

Thanks,
Hitakshi Chauhan

@nikolailic @fabricb @MSFabric @Fabric 

[HitakshiChauhan · Accepted solution]

Issue:
While running notebook code in Microsoft Fabric to display a Lakehouse table, an “Unexpected Error” appears. After stopping the current session and starting a new session, the same code runs successfully.

Root Cause:

Based on analysis, the issue is most likely related to Spark session instability or state corruption, not the code itself.

Possible technical reasons:

1. Spark session state corruption – The active Spark session may enter an inconsistent state.

2. Driver memory/resource exhaustion – Long-running sessions may consume memory or executor resources.

3. Idle timeout or session expiration – Session partially expires but UI still shows it as active.

4. Metadata/cache inconsistency – Cached table metadata may become stale.

5. Background cluster restart – Spark backend may restart but notebook session doesn’t fully reconnect.

Since restarting the session fixes the issue without code changes, it confirms the problem is session-level, not logic-level.

If you face this issue:

1. Stop the current notebook session.

2. Start a new Spark session.

3. Re-run the notebook.

Additional preventive checks:

• Avoid keeping sessions idle for long periods.

• Clear cache before re-running:  Code: spark.catalog.clearCache()

• Restart session after heavy transformations.

• Monitor Spark pool memory usage.

If the issue happens frequently:

• Check Spark pool auto-scale settings.

• Review runtime version compatibility.

• Open a Microsoft support ticket with session logs.

Thanks,
Hitakshi Chauhan

@Fabric @MSFabric @community @fabriccommunity

[HitakshiChauhan]

Hi,

Yes it is solved now And I have upload RAC and Solution here how is solved it.
RCA & Solution: https://community.fabric.microsoft.com/t5/Data-Engineering/RCA-amp-Solution-Unexpected-Error-in-Fabr...

[HitakshiChauhan · Accepted solution]

Issue:
While running notebook code in Microsoft Fabric to display a Lakehouse table, an “Unexpected Error” appears. After stopping the current session and starting a new session, the same code runs successfully.

Root Cause:

Based on analysis, the issue is most likely related to Spark session instability or state corruption, not the code itself.

Possible technical reasons:

1. Spark session state corruption – The active Spark session may enter an inconsistent state.

2. Driver memory/resource exhaustion – Long-running sessions may consume memory or executor resources.

3. Idle timeout or session expiration – Session partially expires but UI still shows it as active.

4. Metadata/cache inconsistency – Cached table metadata may become stale.

5. Background cluster restart – Spark backend may restart but notebook session doesn’t fully reconnect.

Since restarting the session fixes the issue without code changes, it confirms the problem is session-level, not logic-level.

If you face this issue:

1. Stop the current notebook session.

2. Start a new Spark session.

3. Re-run the notebook.

Additional preventive checks:

• Avoid keeping sessions idle for long periods.

• Clear cache before re-running:  Code: spark.catalog.clearCache()

• Restart session after heavy transformations.

• Monitor Spark pool memory usage.

If the issue happens frequently:

• Check Spark pool auto-scale settings.

• Review runtime version compatibility.

• Open a Microsoft support ticket with session logs.

Thanks,
Hitakshi Chauhan

@Fabric @MSFabric @community @fabriccommunity

[HitakshiChauhan]

Thank you so much @Tamanchu for you help and solution it really help me 
thanks for your effort 😊

[Tamanchu]

Hi @HitakshiChauhan,

Happy to help! Glad tthe solutioin insigths while testing, feel free to share them here, it may help others facing the same issue.

Best regards 😊

[V-yubandi-msft]

Hi @HitakshiChauhan ,
@Tamanchu , has correctly pointed out the issue and shared a few steps to follow. Have you tried those steps, and did they help? If you are still facing any issues or need additional information, please let us know.

Thanks for your valuable response @Tamanchu .

Regards,

Yugandhar.

[HitakshiChauhan]

Hi @V-yubandi-msft 

Yes i tried all steps and all are use full.

Thanks
Hitakshi Chauhan

[Tamanchu · Accepted solution]

Great @HitakshiChauhan  👍

A few additional tips for anyone facing this "Unexpected Error" in Fabric Notebooks:

1. Check your Spark session state proactively

# Quick health check before heavy operations
print(f"Spark version: {spark.version}")
print(f"Active session: {spark.sparkContext.applicationId}")

2. Use spark.catalog.refreshTable() before display
If you suspect metadata cache issues on a specific table:

spark.catalog.refreshTable("lakehouse_name.table_name")
display(df)

3. Avoid long idle sessions
Fabric Spark sessions can become unstable after extended idle periods. If you step away for 30+ min, it's often faster to restart the session than to debug random errors.

4. Monitor with Spark UI
Check the Spark UI (Monitoring Hub, Spark Application) for memory pressure or failed stages before restarting blindly.

Thanks for sharing this, it's a common pain point that many Fabric users encounter!

[deborshi_nag]

Hello @HitakshiChauhan can you share you code what you've using to display a Lakehouse table? Is there an error stack trace? There was an issue with the .display() function, raised on 23rd Feb 2026, but it has been resolved now. 

[v-aatheeque]

Hi @HitakshiChauhan 
Following up to confirm if the earlier responses addressed your query. If not, please share your questions and we’ll assist further.