Forum Discussion

stomori's avatar
stomori
New Member
1 year ago
Solved

Fabric Notebook disconnects when running a certain code cell

I am working in a notebook in Fabric using PySpark, and have recently run into an error when running a specific code cell (that normally works). My kernel simply disconnects, and I can't proceed with...
  • burakkaragoz's avatar
    1 year ago

    Hi stomori ,

     

    This looks like a session-level failure that happens before your code even starts running — usually tied to Spark kernel startup or resource allocation issues.

    Here’s what might be causing it and what you can try:

    1. Session Timeout or Idle Expiry
      If the notebook was idle for a while, the session might have expired silently. Try restarting the notebook kernel and re-running the cell immediately.

    2. Spark Pool Resource Limits
      Even if your colleague increased the pool size, check if:

      • The max concurrency is being hit (too many notebooks using the same pool)
      • The session quota per user is exceeded
    3. Code Cell Content
      If the cell has heavy operations (e.g. large joins, wide transformations), try:

      • Breaking it into smaller steps
      • Caching intermediate results
      • Logging the Spark plan (df.explain())
    4. Plugin State: Cleanup
      This usually means the session failed during init and Fabric is cleaning up. It could be a transient backend issue — try running the same cell in a new notebook or after a short wait.

    5. Diagnostics
      Since you don’t have admin rights, ask your admin to check:

      • Spark job logs in the Fabric Admin portal
      • Capacity metrics around the time of failure

    Let me know if you want help reviewing the code in that cell — sometimes a small tweak can avoid triggering these session-level errors.

    If my response resolved your query, kindly mark it as the Accepted Solution to assist others. Additionally, I would be grateful for a 'Kudos' if you found my response helpful.