Forum Discussion
Issues running Spark Notebook
Hi um4ndr ,
This error usually happens when the Livy session gets stale or reset while your notebook is still trying to send a command. That’s why you’re seeing:
InvalidHttpRequestToLivy: Submission failed due to error content =["Statement not found"]
Why it happens:
- The Spark session might have timed out due to inactivity.
- There could be a transient backend issue (especially in shared capacity or trial environments).
- If you're running multiple cells quickly, the session might not be ready to accept the next command.
What you can do:
- Add a retry mechanism with a short delay between retries (e.g., time.sleep(5)).
- Before executing a cell, check if the Spark session is still active using:
spark.sparkContext._jsc.sc().isStopped()
- If this happens frequently, consider splitting heavy operations into smaller steps or running them in a fresh session.
If my response resolved your query, kindly mark it as the Accepted Solution to assist others. Additionally, I would be grateful for a 'Kudos' if you found my response helpful.
Hi um4ndr ,
Thanks for reaching out to the Microsoft fabric community forum.
Livy sessions and their statements are not persistent forever.
If your notebook is idle or takes too long to execute a cell, the statement might expire or be cleaned up before it's accessed again.
Network Latency or Disruption:
A brief network glitch between your environment and the Livy server might cause this.
Race Condition or Resource Unavailability:If many users/notebooks are sharing a Spark pool or compute cluster, Livy might not process requests reliably under load.
If your notebook is connecting to a Spark pool that was stopped or in cold start, the first requests might fail before the backend is ready.
workaround:
As you already noticed, waiting and retrying later worked this supports the idea of a temporary availability or latency issue.
Suggested fixes
Increase Livy timeout settings (if configurable):
Look for options like livy.server.session.timeout and livy.server.statement.timeout (in Azure or Fabric, these might be set via notebook/session config).
Ensure retries are not too fast:
If your retry mechanism is immediate, try adding a delay (e.g., exponential backoff) between retries.
Use smaller, quicker Spark jobs for first cell:
Add a small warm-up job (e.g., spark.range(10).count()) at the beginning to keep the session active and ensure cluster readiness.
Cluster autoscaling or pool warm-up:
If using Azure Synapse / Fabric, make sure your Spark pool is kept warm if needed, or pre-allocate resources to reduce cold start delays.
Check backend service health (Livy logs / Fabric logs):
If this is a recurring issue, consider looking into cluster logs or raising a ticket with Microsoft if you're using Fabric.
Run a lightweight job to keep session active
spark.range(1, 10).count()Handle Livy Errors on Apache Spark in Synapse - Azure Synapse Analytics | Microsoft Learn
Spark Livy Session Timeout Issue - Microsoft Q&A
Submit Spark session jobs using the Livy API - Microsoft Fabric | Microsoft Learn
If this post helped resolve your issue, please consider giving it Kudos and marking it as the Accepted Solution. This not only acknowledges the support provided but also helps other community members find relevant solutions more easily.
We appreciate your engagement and thank you for being an active part of the community.
Best regards,
LakshmiNarayana.
Hello burakkaragoz,
After analyzing the proposed solutions, was chosed the following solution "Increase Livy timeout settings" but due the fact that I'm not using default environment I'll need to make the necessary changes there, but i'm really understand wicht of them it suits me better.
By the way, this problem has not returned yet 🙂